Resurrecting the Site

Last year my blog got infected with a virus and rather than pay my hosting provider to clean it, I decided to turn it off.  Unfortunately I used to use it for reference because I knew I had some random process documented.  Some of the guys at work also had it bookmarked to reference historical articles about some custom piece of code or full stack site that I created.  I also wanted to start writing again to document my studies.

I took a backup of the site from my old hosting provider, but it would not import into WordPress.  I also had a DB dump of the site, so I wrote this program to extract my old posts.  I will slowly re-post my old articles.  Some posts may look good, others will not.  While I have most of my old graphics I do not have all of them, so some may be missing bits and pieces.  I apologize for that.

The process to get my old posts back was to first understand the DB format, then decide how to go about it.  I actually wrote a couple of different scripts, one that dumped all of the posts with date, title and content into one text file.  The second one was to put each article into their own file.  This made it easier to figure out each post.

One thing that I am still struggling with is the formatting for code. Formatting is so important for code, yet I am still learning how this new interface formats and am fighting to make the code look good. I am loosing the battle, but as I continue to work with this editor I hope to eventually win the war.


#!/usr/bin/perl

# 2016-11-12
# Jud Bishop

#use chainrin_wrd01;
#describe wp_posts;
#select ID from wp_posts;
#select post_date from wp_posts where id=128;
#select post_date, post_title, post_content from wp_posts where id=128;

#Database changed
#MariaDB [chainrin_wrd01]> describe wp_posts;
#+-----------------------+---------------------+------+-----+---------------------+----------------+
#| Field | Type | Null | Key | Default | Extra |
#+-----------------------+---------------------+------+-----+---------------------+----------------+
#| ID | bigint(20) unsigned | NO | PRI | NULL | auto_increment |
#| post_author | bigint(20) unsigned | NO | MUL | 0 | |
#| post_date | datetime | NO | | 0000-00-00 00:00:00 | |
#| post_date_gmt | datetime | NO | | 0000-00-00 00:00:00 | |
#| post_content | longtext | NO | | NULL | |
#| post_title | text | NO | | NULL | |
#| post_excerpt | text | NO | | NULL | |
#| post_status | varchar(20) | NO | | publish | |
#| comment_status | varchar(20) | NO | | open | |
#| ping_status | varchar(20) | NO | | open | |
#| post_password | varchar(20) | NO | | | |
#| post_name | varchar(200) | NO | MUL | | |
#| to_ping | text | NO | | NULL | |
#| pinged | text | NO | | NULL | |
#| post_modified | datetime | NO | | 0000-00-00 00:00:00 | |
#| post_modified_gmt | datetime | NO | | 0000-00-00 00:00:00 | |
#| post_content_filtered | longtext | NO | | NULL | |
#| post_parent | bigint(20) unsigned | NO | MUL | 0 | |
#| guid | varchar(255) | NO | | | |
#| menu_order | int(11) | NO | | 0 | |
#| post_type | varchar(20) | NO | MUL | post | |
#| post_mime_type | varchar(100) | NO | | | |
#| comment_count | bigint(20) | NO | | 0 | |
#+-----------------------+---------------------+------+-----+---------------------+----------------+
#

use strict;
use warnings;
use DBI;

my $dbh;
my $sql;
my $sth;
my $fh; #file handle

sub dbi_connect {
 $dbh = DBI->connect('dbi:mysql:dbname=chainrin_wrd01;host=127.0.0.1','chainring','',{AutoCommit=>1,RaiseError=>1,PrintError=>1}) || die "Error connecting: '$DBI::errstr'";
}

sub dbi_disconnect{
      $sth->finish;
      $dbh->disconnect;
}

sub sql_prepare {
     print "$sql\n";
     $sth = $dbh->prepare($sql) || die "Error preparing: $DBI::errstr";
}

sub sql_table_print {

my $result = $sth->execute || die "Error executing: $DBI::errstr";

# HEADER
 print "Field names: @{ $sth->{NAME} }\n";

# DATA
 while (my @data = $sth->fetchrow_array()) {
 my $date = $data[0];
 $date =~ s/\r//g;
 my $title = $data[1];
 $title =~ s/\r//g;
 my $content = $data[2];
 $content =~ s/\r//g;

# It's not pretty, but it's legible.
 my $filename = $title;
 $filename =~ s/\ /-/g;
 $filename =~ s/:/-/g;
 $filename =~ s/\>/-/g;
 $filename =~ s/\</-/g;
 $filename =~ s/\//-/g;

 open_file($filename);
   print $fh "$date\n";
   print $fh "$title\n";
   print $fh "$content\n";
   print $fh "\n";
 close_file();
 }

}

sub open_file {
 print "open_file\n";
 my $filename = shift;
 if ($filename eq ''){ $filename = "filename"; }
 print "$filename\n";
 $filename = "/tmp/Posts/" . $filename;
 open($fh, '>', $filename) || die "Unable to open file: $!";
}

sub close_file {
 close ($fh) || die "Unable to close file: $!";
}

# Main
dbi_connect();
$sql = "select post_date, post_title, post_content from wp_posts";
sql_prepare();
sql_table_print();
dbi_disconnect();

 

Posted in Uncategorized | Leave a comment

Hello World

Hello world!

Posted in Uncategorized | Leave a comment

Fighting Multipath

2013-11-19 08:44:59

[root@chevelle ~]# cat /etc/multipath/bindings
# Multipath bindings, Version : 1.0
# NOTE: this file is automatically maintained by the multipath program.
# You should not need to edit this file in normal circumstances.
#
# Format:
# alias wwid
#
mpath0 36a4badb021d20600133389a784a85226
mpath1 36a4badb000291e140000064f3aa78999
mpath2 36a4badb0002b75c6000006334bea77a3
mpath3 36a4badb000291e14000006983aa7b02e
mpath4 36a4badb0002b75c6000006364bea78a5
mpath5 36a4badb021d32c00132d9598938212dc
mpath6 36a4badb000291e1400001704411a6737
mpath7 36a4badb000291e1400001700411a636d
mpath8 36a4badb000291e1400001706411a67a7
mpath9 36a4badb000291e1400001702411a6421
[root@chevelle ~]# multipath -ll
mpath9 (36a4badb000291e1400001702411a6421) dm-4 DELL,MD3000
[size=136G][features=3 queue_if_no_path pg_init_retries 50][hwhandler=1 rdac][rw]
\_ round-robin 0 [prio=100][active]
\_ 2:0:0:3 sdj 8:144 [active][ready]
\_ round-robin 0 [prio=0][enabled]
\_ 1:0:0:3 sde 8:64 [active][ghost]
mpath8 (36a4badb000291e1400001706411a67a7) dm-3 DELL,MD3000
[size=136G][features=3 queue_if_no_path pg_init_retries 50][hwhandler=1 rdac][rw]
\_ round-robin 0 [prio=100][active]
\_ 2:0:0:2 sdi 8:128 [active][ready]
\_ round-robin 0 [prio=0][enabled]
\_ 1:0:0:2 sdd 8:48 [active][ghost]
mpath7 (36a4badb000291e1400001700411a636d) dm-2 DELL,MD3000
[size=10M][features=3 queue_if_no_path pg_init_retries 50][hwhandler=1 rdac][rw]
\_ round-robin 0 [prio=100][active]
\_ 2:0:0:1 sdh 8:112 [active][ready]
\_ round-robin 0 [prio=0][enabled]
\_ 1:0:0:1 sdc 8:32 [active][ghost]
mpath6 (36a4badb000291e1400001704411a6737) dm-1 DELL,MD3000
[size=10M][features=3 queue_if_no_path pg_init_retries 50][hwhandler=1 rdac][rw]
\_ round-robin 0 [prio=100][active]
\_ 2:0:0:0 sdg 8:96 [active][ready]
\_ round-robin 0 [prio=0][enabled]
\_ 1:0:0:0 sdb 8:16 [active][ghost]
[root@chevelle ~]# man mkqdisk
[root@chevelle ~]# mkqdisk -L
mkqdisk v0.6.0
/dev/dm-5:
/dev/mapper/mpath6p1:
/dev/mpath/mpath6p1:
Magic: eb7a62c2
Label: qdisk
Created: Thu Jun 3 18:40:33 2010
Host: chevelle
Kernel Sector Size: 512
Recorded Sector Size: 512

[root@chevelle ~]# fdisk -l

Disk /dev/sda: 146.1 GB, 146163105792 bytes
255 heads, 63 sectors/track, 17769 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

Device Boot Start End Blocks Id System
/dev/sda1 * 1 128 1020127+ 83 Linux
Partition 1 does not end on cylinder boundary.
/dev/sda2 128 651 4200997+ 82 Linux swap / Solaris
Partition 2 does not end on cylinder boundary.
/dev/sda3 651 17769 137500335 8e Linux LVM

Disk /dev/sdf: 20 MB, 20971520 bytes
64 heads, 32 sectors/track, 20 cylinders
Units = cylinders of 2048 * 512 = 1048576 bytes

Disk /dev/sdf doesn’t contain a valid partition table

Disk /dev/sdg: 10 MB, 10485760 bytes
255 heads, 63 sectors/track, 1 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

Device Boot Start End Blocks Id System
/dev/sdg1 1 1 8001 83 Linux

Disk /dev/sdh: 10 MB, 10485760 bytes
64 heads, 32 sectors/track, 10 cylinders
Units = cylinders of 2048 * 512 = 1048576 bytes

Disk /dev/sdh doesn’t contain a valid partition table

Disk /dev/sdi: 146.2 GB, 146267963392 bytes
255 heads, 63 sectors/track, 17782 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

Disk /dev/sdi doesn’t contain a valid partition table

Disk /dev/sdj: 146.2 GB, 146267963392 bytes
255 heads, 63 sectors/track, 17782 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

Disk /dev/sdj doesn’t contain a valid partition table

Disk /dev/sdk: 20 MB, 20971520 bytes
64 heads, 32 sectors/track, 20 cylinders
Units = cylinders of 2048 * 512 = 1048576 bytes

Disk /dev/sdk doesn’t contain a valid partition table

Disk /dev/dm-1: 10 MB, 10485760 bytes
255 heads, 63 sectors/track, 1 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

Device Boot Start End Blocks Id System
/dev/dm-1p1 1 1 8001 83 Linux

Disk /dev/dm-2: 10 MB, 10485760 bytes
255 heads, 63 sectors/track, 1 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

Disk /dev/dm-2 doesn’t contain a valid partition table

Disk /dev/dm-3: 146.2 GB, 146267963392 bytes
255 heads, 63 sectors/track, 17782 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

Disk /dev/dm-3 doesn’t contain a valid partition table

Disk /dev/dm-4: 146.2 GB, 146267963392 bytes
255 heads, 63 sectors/track, 17782 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

Disk /dev/dm-4 doesn’t contain a valid partition table

Disk /dev/dm-5: 8 MB, 8193024 bytes
255 heads, 63 sectors/track, 0 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

Disk /dev/dm-5 doesn’t contain a valid partition table
[root@chevelle ~]# mount
/dev/mapper/vg00-root on / type ext3 (rw)
proc on /proc type proc (rw)
sysfs on /sys type sysfs (rw)
devpts on /dev/pts type devpts (rw,gid=5,mode=620)
/dev/mapper/vg00-var on /var type ext3 (rw)
/dev/mapper/vg00-usr on /usr type ext3 (rw)
/dev/mapper/vg00-usrlocal on /usr/local type ext3 (rw)
/dev/mapper/vg00-home on /home type ext3 (rw)
/dev/mapper/vg00-opt on /opt type ext3 (rw)
/dev/mapper/vg00-tmp on /tmp type ext3 (rw)
/dev/sda1 on /boot type ext3 (rw)
tmpfs on /dev/shm type tmpfs (rw)
/dev/mapper/vg00-lvpatrol on /patrol type ext3 (rw)
/dev/mapper/vg00-clusterlv on /Cluster_Scripts type ext3 (rw)
none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw)
sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw)
//172.22.225.73/nwapex/data on /opt/nwapex type cifs (rw,mand)
//172.22.100.127/mirth on /opt/mirth type cifs (rw,mand)
//172.22.111.87/data on /opt/bn1 type cifs (rw,mand)
//172.22.225.130/kronos/InterfaceDesigner/Interface Source Files on /opt/kronos type cifs (rw,mand)
//172.22.41.201/Company on /opt/proscript type cifs (rw,mand)
//172.22.100.244/ASD on /opt/murphy type cifs (rw,mand)
//172.22.100.252/StarData on /opt/epsi type cifs (rw,mand)
nfsd on /proc/fs/nfsd type nfsd (rw)
none on /sys/kernel/config type configfs (rw)
/dev/mapper/hbovg-hbo on /hbo type ext3 (rw)
/dev/mapper/hbovg-hboc on /hboc type ext3 (rw)
/dev/mapper/hbovg-mis on /mis type ext3 (rw)
/dev/mapper/hbovg-temphbo on /temphbo type ext3 (rw)
[root@chevelle ~]# ls -l /dev/disk/by-uuid
total 0
lrwxrwxrwx 1 root root 10 Nov 13 03:47 8ebe73bc-0939-401d-b3e4-1d193e433abe -> ../../sda1
[root@chevelle ~]# ls -l /dev/disk/by-
by-id/ by-label/ by-path/ by-uuid/
[root@chevelle ~]# ls -l /dev/disk/by-path/
total 0
lrwxrwxrwx 1 root root 9 Nov 13 03:47 pci-0000:00:1f.2-scsi-0:0:0:0 -> ../../sr0
lrwxrwxrwx 1 root root 9 Nov 13 03:47 pci-0000:03:00.0-scsi-0:2:0:0 -> ../../sda
lrwxrwxrwx 1 root root 10 Nov 13 03:47 pci-0000:03:00.0-scsi-0:2:0:0-part1 -> ../../sda1
lrwxrwxrwx 1 root root 10 Nov 13 03:47 pci-0000:03:00.0-scsi-0:2:0:0-part2 -> ../../sda2
lrwxrwxrwx 1 root root 10 Nov 13 03:47 pci-0000:03:00.0-scsi-0:2:0:0-part3 -> ../../sda3
lrwxrwxrwx 1 root root 9 Nov 13 03:47 pci-0000:08:08.0-sas-0x50026b9139522b00:4:0-0x5a4badb42b75c60c:0 -> ../../sdc
lrwxrwxrwx 1 root root 9 Nov 13 03:47 pci-0000:0a:08.0-sas-0x50026b9139525800:4:0-0x5a4badb4291e140c:0 -> ../../sdg
lrwxrwxrwx 1 root root 10 Nov 13 03:47 pci-0000:0a:08.0-sas-0x50026b9139525800:4:0-0x5a4badb4291e140c:0-part1 -> ../../sdg1
[root@chevelle ~]# ls -l /dev/disk/by-
by-id/ by-label/ by-path/ by-uuid/
[root@chevelle ~]# ls -l /dev/disk/by-label/
total 0
lrwxrwxrwx 1 root root 10 Nov 13 03:47 boot -> ../../sda1
[root@chevelle ~]# ls -l /dev/disk/by-id/
total 0
lrwxrwxrwx 1 root root 9 Nov 13 03:47 scsi-36a4badb000291e1400001700411a636d -> ../../sdc
lrwxrwxrwx 1 root root 9 Nov 13 03:47 scsi-36a4badb000291e1400001702411a6421 -> ../../sde
lrwxrwxrwx 1 root root 9 Nov 13 03:47 scsi-36a4badb000291e1400001704411a6737 -> ../../sdb
lrwxrwxrwx 1 root root 10 Nov 13 03:47 scsi-36a4badb000291e1400001704411a6737-part1 -> ../../sdg1
lrwxrwxrwx 1 root root 9 Nov 13 03:47 scsi-36a4badb000291e1400001706411a67a7 -> ../../sdd
lrwxrwxrwx 1 root root 9 Nov 13 03:47 scsi-36a4badb0002b75c6000015fa525d18aa -> ../../sdf
lrwxrwxrwx 1 root root 9 Nov 13 03:47 scsi-36a4badb021d32c00132d9598938212dc -> ../../sda
lrwxrwxrwx 1 root root 10 Nov 13 03:47 scsi-36a4badb021d32c00132d9598938212dc-part1 -> ../../sda1
lrwxrwxrwx 1 root root 10 Nov 13 03:47 scsi-36a4badb021d32c00132d9598938212dc-part2 -> ../../sda2
lrwxrwxrwx 1 root root 10 Nov 13 03:47 scsi-36a4badb021d32c00132d9598938212dc-part3 -> ../../sda3
[root@chevelle ~]# ls -l /dev/disk/by-label/
total 0
lrwxrwxrwx 1 root root 10 Nov 13 03:47 boot -> ../../sda1
[root@chevelle ~]# ls -l /dev/disk/by-path/
total 0
lrwxrwxrwx 1 root root 9 Nov 13 03:47 pci-0000:00:1f.2-scsi-0:0:0:0 -> ../../sr0
lrwxrwxrwx 1 root root 9 Nov 13 03:47 pci-0000:03:00.0-scsi-0:2:0:0 -> ../../sda
lrwxrwxrwx 1 root root 10 Nov 13 03:47 pci-0000:03:00.0-scsi-0:2:0:0-part1 -> ../../sda1
lrwxrwxrwx 1 root root 10 Nov 13 03:47 pci-0000:03:00.0-scsi-0:2:0:0-part2 -> ../../sda2
lrwxrwxrwx 1 root root 10 Nov 13 03:47 pci-0000:03:00.0-scsi-0:2:0:0-part3 -> ../../sda3
lrwxrwxrwx 1 root root 9 Nov 13 03:47 pci-0000:08:08.0-sas-0x50026b9139522b00:4:0-0x5a4badb42b75c60c:0 -> ../../sdc
lrwxrwxrwx 1 root root 9 Nov 13 03:47 pci-0000:0a:08.0-sas-0x50026b9139525800:4:0-0x5a4badb4291e140c:0 -> ../../sdg
lrwxrwxrwx 1 root root 10 Nov 13 03:47 pci-0000:0a:08.0-sas-0x50026b9139525800:4:0-0x5a4badb4291e140c:0-part1 -> ../../sdg1
[root@chevelle ~]# ls -l /dev/disk/by-uuid/
total 0
lrwxrwxrwx 1 root root 10 Nov 13 03:47 8ebe73bc-0939-401d-b3e4-1d193e433abe -> ../../sda1
[root@chevelle ~]# ls /dev/mapper/
control hbovg-h1686n1.v01a hbovg-h1686n1.v06a hbovg-h1686n1.v11a hbovg-hbo mpath6p1 vg00-home vg00-usr
hbovg-h1686n1.bila hbovg-h1686n1.v02a hbovg-h1686n1.v07a hbovg-h1686n1.v12a hbovg-hboc mpath7 vg00-lvpatrol vg00-usrlocal
hbovg-h1686n1.jn1a hbovg-h1686n1.v03a hbovg-h1686n1.v08a hbovg-h1686n1.v13a hbovg-mis mpath8 vg00-opt vg00-var
hbovg-h1686n1.jn2a hbovg-h1686n1.v04a hbovg-h1686n1.v09a hbovg-h1686n1.v14a hbovg-temphbo mpath9 vg00-root
hbovg-h1686n1.v00a hbovg-h1686n1.v05a hbovg-h1686n1.v10a hbovg-h1686n1.v15a mpath6 vg00-clusterlv vg00-tmp
[root@chevelle ~]# blkid
/dev/mapper/vg00-tmp: LABEL=”/tmp” UUID=”7f389f25-cd20-4b24-ac68-04e9af0ebd04″ TYPE=”ext3″
/dev/mapper/vg00-opt: LABEL=”/opt” UUID=”97e5e00c-0ac4-4821-8248-1cba50920e9b” TYPE=”ext3″
/dev/mapper/vg00-home: LABEL=”/home” UUID=”849f2a65-05a6-42b1-af8c-7ead1b33fb9f” TYPE=”ext3″
/dev/mapper/vg00-usrlocal: LABEL=”/usr/local” UUID=”d0c46c4c-2a2c-4c4b-a1b8-d8a83498b5d9″ TYPE=”ext3″
/dev/mapper/vg00-usr: LABEL=”/usr” UUID=”64887754-c0bc-442b-9b48-f785aa5a0c5c” TYPE=”ext3″
/dev/mapper/vg00-var: LABEL=”/var” UUID=”fed3d412-6b77-4014-b8a9-17471c922399″ TYPE=”ext3″
/dev/mapper/vg00-root: LABEL=”/” UUID=”19e1edd4-ac66-4c2e-8c26-2a8555539b65″ TYPE=”ext3″
/dev/sda2: TYPE=”swap”
/dev/sda1: LABEL=”/boot” UUID=”8ebe73bc-0939-401d-b3e4-1d193e433abe” TYPE=”ext3″
/dev/vg00/root: UUID=”19e1edd4-ac66-4c2e-8c26-2a8555539b65″ TYPE=”ext3″ LABEL=”/”
/dev/scd0: LABEL=”MD3000_2.2.0.17″ TYPE=”iso9660″
/dev/mapper/vg00-clusterlv: UUID=”485aea89-b699-49d5-8d87-bae50f80c9e7″ TYPE=”ext3″
/dev/mapper/vg00-lvpatrol: LABEL=”/patrol” UUID=”86ff22dd-d895-4a2f-beca-f3cc8b5e7bd0″ TYPE=”ext3″
/dev/dvd: LABEL=”MD3000_2.2.0.17″ TYPE=”iso9660″
/dev/sr0: LABEL=”MD3000_2.2.0.17″ TYPE=”iso9660″
/dev/mapper/hbovg-hbo: UUID=”cf90e0da-30b7-41b6-a73d-5e39bdddd013″ TYPE=”ext3″
/dev/mapper/hbovg-hboc: UUID=”43cade5f-ab81-4a7d-b134-ad0917c999e3″ TYPE=”ext3″
/dev/mapper/hbovg-mis: UUID=”14265200-9022-496b-bc6f-8ae3e00c3f13″ TYPE=”ext3″
/dev/mapper/hbovg-temphbo: UUID=”8f1af44c-2207-49ed-91f9-c44283794713″ TYPE=”ext3″
[root@chevelle ~]#

Posted in Linux | Leave a comment

Storage Pod

At the Circus we just built our first Backblaze storage pod and I would like to take the time to document it. We rebuilt the server a number of times for testing and verification with different numbers of disks so output may differ throughout this post.

The cost per terabyte is right up our alley as we are a non-profit hospital. We tried to set ours up as a Windows server so it would have direct attached storage but changed direction and decided to make it a Linux based iSCSI target.

Disk Mapping
The first problem is mapping out the port multiplier backplanes. If you follow this link it shows the way the pod is supposed to be built, however, our drives did not map out accordingly. We took the time to map out our drives by literally shutting down, pulling a disk and turning the server back on to find the layout. If you don’t take the time to do this, I feel for you when a disk dies and you try to figure out how to replace it.

Boot Drives.
sd 0:0:0:0: [sda]
sd 1:0:0:0: [sdb]

First row from right.
sd 7:0:0:0: [sdh]
sd 7:1:0:0: [sdi]
sd 7:2:0:0: [sdj]
sd 7:3:0:0: [sdk]
sd 7:4:0:0: [sdl]

sd 6:0:0:0: [sdc]
sd 6:1:0:0: [sdd]
sd 6:2:0:0: [sde]
sd 6:3:0:0: [sdf]
sd 6:4:0:0: [sdg]

sd 8:0:0:0: [sdm]
sd 8:1:0:0: [sdn]
sd 8:2:0:0: [sdo]
sd 8:3:0:0: [sdp]
sd 8:4:0:0: [sdq

Second row from right.
sd 11:0:0:0: [sdw]
sd 11:1:0:0: [sdx]
sd 11:2:0:0: [sdy]
sd 11:3:0:0: [sdz]
sd 11:4:0:0: [sdaa]

sd 10:0:0:0: [sdr]
sd 10:1:0:0: [sds]
sd 10:2:0:0: [sdt]
sd 10:3:0:0: [sdu]
sd 10:4:0:0: [sdv]

sd 12:0:0:0: [sdab]
sd 12:1:0:0: [sdac]
sd 12:2:0:0: [sdad]
sd 12:3:0:0: [sdae]
sd 12:4:0:0: [sdaf]

Third row from right.
sd 14:0:0:0: [sdag]
sd 14:1:0:0: [sdah]
sd 14:2:0:0: [sdai]
sd 14:3:0:0: [sdaj]
sd 14:4:0:0: [sdak]

sd 15:0:0:0: [sdal]
sd 15:1:0:0: [sdam]
sd 15:2:0:0: [sdan]
sd 15:3:0:0: [sdao]
sd 15:4:0:0: [sdap]

sd 16:0:0:0: [sdaq]
sd 16:1:0:0: [sdar]
sd 16:2:0:0: [sdas]
sd 16:3:0:0: [sdat]
sd 16:4:0:0: [sdau]

Disk Setup
The next problem you have is that fdisk will not handle partitions larger than 2TB, parted to the rescue. Because there were forty-five 4TB disks in the server I did not want to have to do it manually. The other problem was that we had also tested the server as a Windows server so it already had partitions on the disks. As a result we had to remove the old partitions, then create a new one. Luckily you can script parted. Please note that parts of the script are commented out because we ran the script multiple times for different setups.

for I in `dmesg|grep ^sd|cut -d \  -f 1,2,3|grep -v Attach |sort -u | cut -d [ -f 2 | cut -d ] -f 1 `; do echo /dev/${I}\ ; done >>devices-list.txt

cat /usr/local/bin/parted-script.sh 
#!/bin/sh
for i in `cat devices-list.txt`
do
# delete previous partitions
#parted $i --script -- rm 1
#parted $i --script -- rm 2
#parted $i --script -- rm 3

# create partition to take whole disk
parted $i --script -- mkpart primary ext4 1 -1

# set type lvm for jbod
# parted $i --script -- set 1 lvm on

# set type RAID for RAID 6.
parted $i --script -- set 1 raid on

parted $i --script print
done

Create the RAID
The first time through we made all of the disks a JBOD to play, but long term that did not make sense. As a result I am only going to document creating a RAID 6 iSCSI target for Windows servers as this is the purpose of our storage pod.

I try not to do many tasks manually, so here is the work around for trying not to have type 45 disk names.

dmesg|grep ^sd|cut -d \  -f 1,2,3|grep -v Attach |sort -u | cut -d [ -f 2 | cut -d ] -f 1 >>devices.txt
for I in `cat devices.txt`; do  echo -n /dev/${I}1\ ; done >devices1.txt

This creates a file with all of the disk names.

cat devices1.txt 
/dev/sda1 /dev/sdc1 /dev/sdb1 /dev/sdd1 /dev/sde1 /dev/sdf1 /dev/sdg1 /dev/sdh1 /dev/sdi1 /dev/sdj1 /dev/sdk1 /dev/sdl1 /dev/sdm1 /dev/sdn1 /dev/sdo1 /dev/sdp1 /dev/sdq1 /dev/sdr1 /dev/sds1 /dev/sdt1 /dev/sdu1 /dev/sdv1 /dev/sdw1 /dev/sdx1 /dev/sdy1 /dev/sdz1 /dev/sdaa1 /dev/sdab1 /dev/sdac1 /dev/sdad1 /dev/sdae1 /dev/sdaf1 

Create the different software RAID configurations. I created three RAID devices, md0, md1 and md2.

This mdadm command creates a RAID 6 container with 14 physical disks and one spare. We were being cautious with our data.

mdadm --create --verbose /dev/md1 --level=6 --chunk=512 --raid-devices=14 --spare-devices=1 /dev/sdr1 /dev/sds1 /dev/sdt1 /dev/sdu1 /dev/sdv1 /dev/sdw1 /dev/sdx1 /dev/sdy1 /dev/sdz1 /dev/sdaa1 /dev/sdab1

This mdadm command create a RAID 6 container with all 15 physical disks, I used this configuration for testing the throughput later.


mdadm --create --verbose /dev/md0 --level=6 --chunk=512 --raid-devices=15 /dev/sdc1 /dev/sdd1 /dev/sde1 /dev/sdf1 /dev/sdh1 /dev/sdg1 /dev/sdi1 /dev/sdk1 /dev/sdj1 /dev/sdl1 /dev/sdm1 /dev/sdn1 /dev/sdo1 /dev/sdp1 /dev/sdq1 

mdadm --create --verbose /dev/md1 --level=6 --chunk=512 --raid-devices=15 /dev/sdr1 /dev/sds1 /dev/sdt1 /dev/sdu1 /dev/sdv1 /dev/sdw1 /dev/sdx1 /dev/sdy1 /dev/sdz1 /dev/sdaa1 /dev/sdab1 /dev/sdad1 /dev/sdac1 /dev/sdae1 

/mdadm --create --verbose /dev/md2 --level=6 --chunk=512 --raid-devices=15 /dev/sdag1 /dev/sdah1 /dev/sdai1 /dev/sdaj1 /dev/sdak1 /dev/sdal1 /dev/sdam1 /dev/sdan1 /dev/sdao1 /dev/sdap1 /dev/sdaq1 /dev/sdar1 /dev/sdas1 /dev/sdat1 /dev/sdau1 

If you are truly just building an iSCSI target the next steps are pointless. I wanted to do a throughput test so I had to lay down a file system, but once again there were problems. There is a 16TB size limit with mke2fs that ships with RedHat, as a result you need to build a newer version of e2fsprogs.

git clone git://git.kernel.org/pub/scm/fs/ext2/e2fsprogs.git
cd e2fsprogs
mkdir build ; cd build/
../configure
make
make install

mke2fs -O 64bit,has_journal,extents,huge_file,flex_bg,uninit_bg,dir_nlink,extra_isize -i 4194304 /dev/md0
mke2fs 1.43-WIP (22-Sep-2012)

Warning: the fs_type huge is not defined in mke2fs.conf

Filesystem label=
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
Stride=0 blocks, Stripe width=0 blocks
11446336 inodes, 11721045504 blocks
586052275 blocks (5.00%) reserved for the super user
First data block=0
Maximum filesystem blocks=13870563328
357698 block groups
32768 blocks per group, 32768 fragments per group
32 inodes per group
Superblock backups stored on blocks: 
	32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208, 
	4096000, 7962624, 11239424, 20480000, 23887872, 71663616, 78675968, 
	102400000, 214990848, 512000000, 550731776, 644972544, 1934917632, 
	2560000000, 3855122432, 5804752896

Allocating group tables: done                            
Writing inode tables: done                            
Creating journal (32768 blocks): done
Writing superblocks and filesystem accounting information: done         

Next is mount it up and test.

mount -t ext4 /dev/md0 /backup0

mount
/dev/mapper/vg_leroy-lv_root on / type ext4 (rw)
proc on /proc type proc (rw)
sysfs on /sys type sysfs (rw)
devpts on /dev/pts type devpts (rw,gid=5,mode=620)
tmpfs on /dev/shm type tmpfs (rw)
/dev/mapper/ddf1_Rootp1 on /boot type ext4 (rw)
/dev/mapper/vg_leroy-lv_home on /home type ext4 (rw)
none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw)
sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw)
/dev/md0 on /backup0 type ext4 (rw)

watch cat /proc/mdstat 

Every 2.0s: cat /proc/mdstat                                                                                                                                                              Tue Nov 13 14:50:58 2012
md2 : active raid6 sdau1[14] sdat1[13] sdas1[12] sdar1[11] sdaq1[10] sdap1[9] sdao1[8] sdan1[7] sdam1[6] sdal1[5] sdak1[4] sdaj1[3] sdai1[2] sdah1[1] sdag1[0]
      50791197184 blocks super 1.2 level 6, 512k chunk, algorithm 2 [15/15] [UUUUUUUUUUUUUUU]
      [>....................]  resync =  0.0% (72704/3907015168) finish=5372.5min speed=12117K/sec
      
md1 : active raid6 sdaf1[14] sdae1[13] sdac1[12] sdad1[11] sdab1[10] sdaa1[9] sdz1[8] sdy1[7] sdx1[6] sdw1[5] sdv1[4] sdu1[3] sdt1[2] sds1[1] sdr1[0]
      50791197184 blocks super 1.2 level 6, 512k chunk, algorithm 2 [15/15] [UUUUUUUUUUUUUUU]
      [>....................]  resync =  0.0% (2583680/3907015168) finish=4776.4min speed=13623K/sec
      
md0 : active raid6 sdq1[14] sdp1[13] sdo1[12] sdn1[11] sdm1[10] sdl1[9] sdj1[8] sdk1[7] sdi1[6] sdg1[5] sdh1[4] sdf1[3] sde1[2] sdd1[1] sdc1[0]
      50791197184 blocks super 1.2 level 6, 512k chunk, algorithm 2 [15/15] [UUUUUUUUUUUUUUU]
      [>....................]  resync =  0.0% (3255892/3907015168) finish=5886.7min speed=11052K/sec

Finally you need to save the software raid configuration.

mdadm --detail --scan >> /etc/mdadm.conf

Testing
I wanted to try a throughput test so I copied a CD over to the server. We just weren’t getting enough throughput with the reads and writes so I decided to create a ramdisk, read from it and write to the filesystem.

Create the ramdisk.

ls -alh /dev/ram*
mknod -m 660 /dev/ramdisk b 1 1
chown root.disk /dev/ramdisk
dd if=/dev/zero of=/dev/ramdisk bs=1k count=4194304
/sbin/mkfs -t ext2 -m 0 /dev/ramdisk 16384
mkdir /ramdisk
mount -t ext2 /dev/ramdisk /ramdisk
dd if=/dev/urandom of=/ramdisk/file.txt bs=1k count=15k
ls -alh /ramdisk/

Now copy the 15mb file from the ramdisk 500,000 times. I ran this script for /backup0, /backup1 and /backup2.

for i in `jot -s 1 -e 500000`; do  cp /ramdisk/file.txt /backup0/test0-${i}; done

And the test output, in one minute we had written 12MB.

date && df -h
Sat Nov 10 16:29:49 CST 2012
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/vg_leroy-lv_root
                       50G  3.2G   44G   7% /
tmpfs                 3.9G     0  3.9G   0% /dev/shm
/dev/mapper/ddf1_Rootp1
                      485M   37M  423M   8% /boot
/dev/mapper/vg_leroy-lv_home
                      236G  188M  224G   1% /home
/dev/md0               48T   78G   45T   1% /backup0
/dev/md1               48T   84G   45T   1% /backup1
/dev/md2               48T   78G   45T   1% /backup2
/dev/ramdisk           16M   16M  302K  99% /ramdisk


date && df -h
Sat Nov 10 16:30:49 CST 2012
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/vg_leroy-lv_root
                       50G  3.2G   44G   7% /
tmpfs                 3.9G     0  3.9G   0% /dev/shm
/dev/mapper/ddf1_Rootp1
                      485M   37M  423M   8% /boot
/dev/mapper/vg_leroy-lv_home
                      236G  188M  224G   1% /home
/dev/md0               48T   82G   45T   1% /backup0
/dev/md1               48T   88G   45T   1% /backup1
/dev/md2               48T   82G   45T   1% /backup2
/dev/ramdisk           16M   16M  302K  99% /ramdisk

And the IOSTAT command while it was writing.

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           0.22    0.00    6.40   55.14    0.00   38.23

Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
sda               2.18        73.97       117.05    6527092   10328224
sdb               0.79         0.06       117.05       5536   10328224
sdc              87.69       882.70     82847.18   77890479 7310556800
sdd              86.17       715.18     82822.85   63108394 7308410360
sde              11.11       714.80      6402.75   63074967  564987674
sdf               4.92       714.24       135.85   63025858   11987866
sdh               5.16       714.32       134.02   63032209   11825930
sdg               5.10       714.65       135.50   63062197   11956714
sdi               4.98       714.30       133.72   63030809   11799450
sdk               4.92       714.45       133.54   63044265   11784026
sdj               4.95       714.24       133.88   63025249   11813618
sdl               5.02       714.18       134.05   63020313   11828514
sdm               5.08       714.06       133.96   63009609   11821122
sdn               5.00       714.15       133.74   63017213   11801082
sdo               4.97       714.16       133.85   63018757   11811058
sdp               4.62       714.38       130.34   63038351   11501106
sdq               4.59       714.04       128.39   63007996   11329122
sdr               4.82       784.45        45.74   69221236    4036394
sds               4.78       784.46        46.01   69221608    4060330
sdt               4.79       784.55        47.90   69229964    4226866
sdu               4.79       784.75        46.05   69247648    4063482
sdv               4.75       784.68        45.86   69241532    4046386
sdw               4.81       784.77        45.80   69249556    4041530
sdx               4.78       784.75        45.90   69247718    4050298
sdy               4.76       784.88        45.84   69259062    4045058
sdz               4.77       784.79        45.92   69251000    4052066
sdaa              4.75       784.69        45.99   69242304    4058370
sdab              4.47       784.89        45.81   69259548    4042074
sdad              4.40       784.83        45.80   69254484    4041794
sdac              4.32       784.92        47.64   69262304    4203442
sdae              4.23       784.84        45.64   69255316    4027730
sdaf              4.12       784.88        45.60   69258620    4024146
sdag              4.42       702.93        41.19   62027358    3634226
sdah              4.37       702.73        41.38   62009962    3651746
sdai              4.37       702.79        41.67   62015092    3677450
sdaj              4.35       702.87        41.50   62022040    3661962
sdak              4.35       703.19        41.26   62050556    3640690
sdal              4.37       703.53        40.96   62080556    3614570
sdam              4.34       703.60        40.85   62086828    3604554
sdan              4.33       703.42        41.02   62070532    3620082
sdao              4.34       703.41        41.22   62069532    3637226
sdap              4.32       703.41        41.15   62069548    3631570
sdaq              4.08       703.40        41.29   62069444    3643514
sdar              4.01       703.10        41.58   62042804    3669194
sdas              3.94       702.87        41.55   62021960    3666570
sdat              3.85       703.25        40.92   62055866    3611258
sdau              3.77       703.06        40.93   62039220    3611930
dm-0             16.60        73.91       117.05    6521508   10328224
dm-1              0.05         0.37         0.00      32976        168
dm-2             16.51        73.25       117.04    6463516   10328056
dm-3              5.79        72.21        32.65    6372058    2880648
dm-4             10.60         0.36        84.40      32112    7447384
dm-5              0.04         0.31         0.00      26938         24
md0              54.27         0.56       433.70      49578   38270384
md1              67.95         0.56       543.17      49602   47930328
md2              60.73         0.56       485.41      49594   42832904

Create an iSCSI target.
Once you create the iSCSI target and format the drive with a Windows file system, you have lost any data that was on the drive you created earlier. Remember with iSSCSI you are presenting a target “physical” drive.

Install the iSCSI target utilities.

yum install scsi-target-utils

The iSCSI configuration file.

cat /etc/tgt/targets.conf
default-driver iscsi

# Parameters below are only global. They can't be configured per LUN.
# Only allow connections from 192.168.100.1 and 192.168.200.5
initiator-address 192.168.100.1
initiator-address 192.168.200.5

<target iqn.2012-11.org.eamc:leroy.target0>
	backing-store /dev/md0	
	write-cache off
	lun 11
</target>
<target iqn.2012-11.org.eamc:leroy.target1>
	backing-store /dev/md1
	write-cache off
	lun 12
</target>

Turn on tgtd.

chkconfig iptables off
chkconfig tgtd on
chkconfig tgtd --list

SMARTD
One of the guys on the team brought up that we should be doing some hard drive monitoring to make sure we knew if we were having trouble with a drive. As a result I installed smartmontools and configured the daemon to email when a drive starts to fail.

Install smartmontools.

yum install smartmontools

Edit the configuration file to email, but the first time test to make sure an email is sent.

cat /etc/smartd.conf
DEVICESCAN -a -I 194 -W 4,45,55 -R 5 -m jud@circus.org -M test

Start the smartd daemon.

chkconfig smartd on
service smartd start

Now go back and remove the -M test from the configuration file to make sure you don’t get emails every time the smartd daemon restarts. There are a number of configuration options, so read the /etc/smartd.conf file for a better understanding.

Some random commands:

mdadm --stop /dev/md124
mdadm --remove /dev/md124
mdadm --query --detail /dev/md1
mdadm --detail-platform
mdadm --monitor
mdadm --explain /dev/md0
Posted in Linux | Leave a comment

Congestion Management

First a comment on the structuring of my notes. In order to make them more legible I have started using headings for each section and then bolding the subsections. I believe it makes it easier to read and I do go back and use these notes to study for the actual test.

Queuing Concepts

A queue organizes packets packets waiting to exit an interface, the size of the queue affects delay, jitter and loss.

  • A longer queue decreases the chance of tail drop but increases average delay and typically increases jitter as well.
  • A shorter queue increases the chance of tail drop but decreases the average delay and typically decreases jitter.
  • If the congestion is sustained for long periods of time drops will be just as likely no matter the queue length.

Hardware Queueor TX Ring

If space is available in the hardware queue no output queuing is performed on a packet. It is only with congestion on the hardware queue that software queues are used.

  • Hardware queues always perform FIFO scheduling and cannot be changed.
  • The hardware queue uses one single queue per interface.
  • IOS shortens the hardware queue automatically when a software queue is applied.
  • The hardware queue length can be configured to a different value.

The command show controllers interface shows information about the hardware queue.

R4#sh controllers s0/0/0
Interface Serial0/0/0
Hardware is GT96K
DTE V.35 TX and RX clocks detected.
idb at 0x65EF56B4, driver data structure at 0x65EFCE60
wic_info 0x65EFD484
Physical Port 0, SCC Num 0
MPSC Registers:
MMCR_L=0x000304C0, MMCR_H=0x00000000, MPCR=0x00000000
CHR1=0x00FE007E, CHR2=0x00000000, CHR3=0x0000064A, CHR4=0x00000000
CHR5=0x00000000, CHR6=0x00000000, CHR7=0x00000000, CHR8=0x00000000
CHR9=0x00000000, CHR10=0x00003008
SDMA Registers:
SDC=0x00002201, SDCM=0x00000080, SGC=0x0000C000
CRDP=0x160FEA50, CTDP=0x160FECD0, FTDB=0x160FECD0
Main Routing Register=0x0003FFC0 BRG Conf Register=0x00480000
Rx Clk Routing Register=0x76543288 Tx Clk Routing Register=0x76543219
GPP Registers:
Conf=0x43030002, Io=0x46064250, Data=0x7B7BBDA9, Level=0x180000  
Conf0=0x43030002, Io0=0x46064250, Data0=0x7B7BBDA9, Level0=0x180000  
0 input aborts on receiving flag sequence
0 throttles, 0 enables
0 overruns
0 transmitter underruns
0 transmitter CTS losts
23 rxintr, 28 txintr, 0 rxerr, 0 txerr
52 mpsc_rx, 0 mpsc_rxerr, 0 mpsc_rlsc, 6 mpsc_rhnt, 47 mpsc_rfsc
6 mpsc_rcsc, 0 mpsc_rovr, 0 mpsc_rcdl, 0 mpsc_rckg, 0 mpsc_bper
0 mpsc_txerr, 29 mpsc_teidl, 0 mpsc_tudr, 0 mpsc_tctsl, 0 mpsc_tckg
0 sdma_rx_sf, 0 sdma_rx_mfl, 0 sdma_rx_or, 0 sdma_rx_abr, 0 sdma_rx_no
0 sdma_rx_de, 0 sdma_rx_cdl, 0 sdma_rx_ce, 0 sdma_tx_rl, 0 sdma_tx_ur, 0 sdma_tx_ctsl
0 sdma_rx_reserr, 0 sdma_tx_reserr
0 rx_bogus_pkts, rx_bogus_flag FALSE 
0 sdma_tx_ur_processed

tx_limited = 0(128), errata19 count1 - 0, count2 - 0

In the above listing, see the line tx_limited = 0(128), errata19 count1 – 0, count2 – 0 at the bottom of the output. This hardware queue holds 128 packets and the 0 means the queue size is not limited by a queuing tool on this interface.

Enable priority queuing to change the hardware queue length.

R4#   
R4#conf t
Enter configuration commands, one per line.  End with CNTL/Z.
R4(config)#int s0/0/0
R4(config-if)#priority-group 1
R4(config-if)#do sh controllers s0/0/0
Interface Serial0/0/0
Hardware is GT96K

Output removed for brevity.

tx_limited = 1(2), errata19 count1 - 0, count2 - 0

After enabling priority queuing with the priority-group command you can see that the new length of the hardware queue is (2) and the 1 means the length is limited as a result of queuing being configured.

The hardware queue length can be changed with the tx-ring-limit x command as seen below, this was done with the priority queue still active.

R4(config-if)#tx-ring-limit 50
R4(config-if)#do sh controllers s0/0/0
Interface Serial0/0/0
Hardware is GT96K

Output removed for brevity.

tx_limited = 1(50), errata19 count1 - 0, count2 - 0

Queuing on Inerfaces, Subinterfaces and Virtual Circuits

Traffic is not even placed in a software queue unless the hardware queue is full, however traffic shaping traffic shaping can cause shaping queues to fill even when when there is no congestion on the physical interface. In effect traffic shaping on the sub interfaces creates congestion between the shaping queues and the physical interface software queues. On a physical interface traffic can only leave at the speed of the physical clock rate, similarly packets can only leave a shaping queue at the traffic-shaping rate.
QoS p.263

Scheduling Concepts

For the test we just need to know the basic concepts of FIFO, PQm CQ and MDRR.

FIFO
FIFO uses tail drop to decide when to drop or enqueue packets. As above, the same holds true for FIFO, a longer queue decreases the chance of tail drop but increases average delay and typically increases jitter as well. A shorter queue increases the chance of tail drop but decreases the average delay and typically decreases jitter.

Configuring FIFO actually requires you to turn off all other types of queuing.

From my example above I am still using priority queuing from above.

R4(config-if)#do sh int s0/0/0
Serial0/0/0 is up, line protocol is up 
  Hardware is GT96K Serial
  MTU 1500 bytes, BW 1544 Kbit/sec, DLY 20000 usec, 
     reliability 255/255, txload 1/255, rxload 1/255
  Encapsulation FRAME-RELAY, loopback not set
  Keepalive set (10 sec)
  CRC checking enabled
  LMI enq sent  206, LMI stat recvd 206, LMI upd recvd 0, DTE LMI up
  LMI enq recvd 0, LMI stat sent  0, LMI upd sent  0
  LMI DLCI 1023  LMI type is CISCO  frame relay DTE
  FR SVC disabled, LAPF state down
  Broadcast queue 0/64, broadcasts sent/dropped 4/0, interface broadcasts 1
  Last input 00:00:00, output 00:00:00, output hang never
  Last clearing of "show interface" counters 00:34:20
  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
vvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvv
  Queueing strategy: priority-list 1
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  Output queue (queue priority: size/max/drops):
     high: 0/20/0, medium: 0/40/0, normal: 0/60/0, low: 0/80/0
  5 minute input rate 0 bits/sec, 0 packets/sec
  5 minute output rate 0 bits/sec, 0 packets/sec
     206 packets input, 5846 bytes, 0 no buffer
     Received 0 broadcasts, 0 runts, 0 giants, 0 throttles
     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort
     210 packets output, 3736 bytes, 0 underruns
     0 output errors, 0 collisions, 8 interface resets
     0 unknown protocol drops
     0 output buffer failures, 0 output buffers swapped out
     0 carrier transitions
     DCD=up  DSR=up  DTR=up  RTS=up  CTS=up

So I remove the priority queue and you can see that the interface is back to default, weighted fair queuing.

R4(config-if)#no priority-group 1     
R4(config-if)#do sh int s0/0/0   
Serial0/0/0 is up, line protocol is up 
  Hardware is GT96K Serial

Output removed for brevity.

  Queueing strategy: weighted fair
  Output queue: 0/1000/64/0 (size/max total/threshold/drops)

To make the interface FIFO queuing I have to remove WFQ.

R4(config-if)#no fair-queue 
R4(config-if)#do sh int s0/0/0
Serial0/0/0 is up, line protocol is up 
  Hardware is GT96K Serial

Output removed for brevity.

  Queueing strategy: fifo
  Output queue: 0/40 (size/max)

You can change the default queue length with the command hold-queue x out.

R4(config-if)#hold-queue 20 out
R4(config-if)#do sh int s0/0/0 
Serial0/0/0 is up, line protocol is up 
  Hardware is GT96K Serial

Output removed for brevity.

  Queueing strategy: fifo
  Output queue: 0/20 (size/max)

Priority Queuing (PQ)
With priority queuing the highest priority queues are always serviced first. There are four queues, High, Medium, Normal and Low. If the High queue has a packet it is serviced, if not the the Medium queue is serviced and on down to the Low priority queue. The process always starts back at the High queue. As a result the lower priority queues get starved. This fact makes it an unpopular queuing choice.

Custom Queuing (CQ)
Custom queuing addresses the largest drawback of PQ, servicing all queues even during congestion. It has 16 queues available, implying 16 classification categories. It does not have the option to service one queue first, and does round-robin service on each queue, beginning with the first queue. CQ takes packets from that queue until the total byte count specified for that queue has been met or exceeded. After that queue has been serviced or does not have nay more packets, CQ moves on to the next queue and repeats the process.

The CQ scheduler essentially guarantees the minimum bandwidth for each queue, while allowing queues to have more bandwidth under the right conditions. If 5 queues have been configured with the byte counts of 5,000, 5,000, 10,000, 10,000 and 20,0000 for queues 1 through 5, The percentage bandwidth given to each queues is 10, 10, 20, 20, and 40 percent. But if queue 4 has no traffic over a short period of time, the CQ scheduler moves to another queue. Only queues 1-3 and 5 have packets waiting so the distribution is changed. The queues would receive 12.5, 12.5, 25, 0 and 50 percent of the bandwidth.

The queues are numbered not named and no queue gets better service than another.
QoS p.270

Modified Deficit Round-Robin (MDRR)
MDRR is similar to the CQ scheduler in that it reserves a percentage of link bandwidth for a particular queue. MDRR removes packets from a queue until the quantum value (QV) for that queue has been removed. MDRR repeats the process for every queue in order from 0 through 7. Any extra bytes sent during this process are treated as a deficit and subtracted from the QV for the next pass. As a result MDRR provides an exact bandwidth reservation.

Concepts and Configuration WFQ, CBWFQ and LLQ

Weighted Fair Queuing (WFQ)
WFQ does not allow classification options to be configured. It classifies packets based on flow. A flow consists of all packets that have the same source and destination IP address, and the same source and destination port numbers. WFQ also favor low-volume higher-precedence flows over large-volume lower-precedent flows. Each flow uses a different queue and up to a maximum of 4096 queues per interface. It also uses modified tail-drop. WFQ may be the most deployed QoS tool on Cisco routers so take your time on this section.

WFQ can be seen as being too fair, with many flows WFQ will give some bandwidth to every flow. WFQ is also a poor choice for voice and interactive video traffic because both need low delay and low jitter. By being too fair it can starve voice and video.

Flows
Flows are identified by five items in a packet.

  • Source IP address
  • Destination IP address
  • Transport layer (TCP or UDP)
  • Source port
  • Destination port
  • IP Precedence

Flows are considered to exist only as long as packets from the flow exist. If there is a break in traffic and no packets are in the queue, it is removed. The show queue command tells about the WFQ’s view of a flow.

WFQ Scheduler
WFQ has two goals:
1. To provide fairness among the existing flows, giving each flow an equal amount of bandwidth. With each flow receiving the same bandwidth lower volume flows prosper while higher volume flows suffer.
2. Provide more bandwidth to flows with higher IP precedence values. The “weight” of WFQ is based on precedence. WFQ provides a fair share of the link bandwidth based on each flows precedence, plus one. Precedence 7 flows get 8 times more bandwidth than precedence 0 flow because (7+1)/(0+1) = 8.

When adding packets to the hardware queue WFQ puts the packet with the lowest sequence number (SN) among all of the queues or flows.

SN calculation:
SN = Previous_SN + (weight * new_packet_length)
Weight = 32,384 / (IP_Precedence + 1)

SN = Previous_SN + ((32,384 / (IP_Precedence + 1)) * new_packet_length)

WFQ calculates the SN before adding a packets to its queue and even before the decision is made to drop the packet because it is based on modified tail drop. The formula considers the length of the packet, the weight of the flow, and the previous SN. By considering the packet length the SN calculates a higher SN for larger packets and a lower number for smaller packets. By including the SN of the previous packet, the formula assigns a larger SN to queues that already have a number of packets enqueued.

WFQ always weights the packets based on the first 3 bits of the ToS byte, the Precedence field.

The larger the precedence value, the lower the weight, making the SN smaller and therefore favoring that flow over another with a lower precedence value.

Precedence Weight
0 32384
1 16192
2 10794
3 8096
4 6476
5 5397
6 4626
7 4048

WFQ Drop Policy, Number of Queues and Queue Length
WFQ places and absolute limit on the number of packets in all queues called the hold-queue limit. If a new packet arrives for any queue and the hold-queue limit has been reached, the packet is discarded.

WFQ also places a limit on individual queues called the congestive discard threshold (CDT). If the individual queues CDT has been reached WFQ looks for a packet with a higher calculated SN in all of the queues. If a packet is found with a higher SN it is discarded the this packet is enqueued.

WFQ also keeps eight hidden queues for overhead traffic generated by the router. WFQ uses a very low weight for these hidden queues in order to give precedence to overhead traffic.

WFQ Configuration
IOS uses WFQ by default on all serial interfaces with bandwidths set at T/1 or E/1 speeds and below. To turn on WFQ use the command fair-queue.
To change the hold queue of an interface use the command hold-queue x out.

Class-Based WFQ (CBWFQ)
CBWFQ uses MQC to classify traffic so anything you can match with MQC you can match with CBWFQ. It can reserve a minimum amount of bandwidth for each queue and can give an actual percentage of the bandwidth.

CBWFQ supports both tail drop and WRED. There are 64 queues available in CBWFQ and WRED can be enabled on any of them. WRED works well for less drop-sensitive traffic such as data but is not a good choice for voice and video.

If a packet is not classified in CBWFQ it goes to the class-default queue. Inside the class-default queue CBWFQ can use either FIFO or WFQ. With WFQ it uses the SN calculation within that queue just like WFQ normally does. Using WFQ in the default class is an advantage for CBWFQ because WFQ treats low-volumes flows well and they are likely to be interactive traffic. So with CBWFQ the traffic you know, classify and reserve the proper bandwidth. For traffic you cannot characterize, let it default into the class-default queue where WFQ dynamically applies fairness by using WFQ.

The CBWFQ scheduler gives a percentage of bandwidth to each class based on the configured values, although the algorithm is not published.

Delay and jitter sensitive traffic still suffer with CBWFQ because other queues can still be serviced while those packets wait.

CBWFQ Configuration
This is an example take from QoS p. 298.

    The criteria:

  • All VOIP payload traffic has been marked with DSCP EF, placed in a queue.
  • All other traffic has been marked with DSCP BE, place in a different queue.
  • Give the VOIP trassif 58 kpbs of bandwidth on the link.
  • Use WRED and WFQ in the non-VOIP traffic.
class-map match-all voip-rtp
	match ip ftp 16384 16383

class-map match-all dscp-ef
	match ip dscp 
! This is the input policy-map.
policy-map voip-be
	class voip-rtp
		set ip dscp ef
	class class-default
		set ip dscp 0

! This is the output policy-map.
policy-map queue-on-dscp
	class descp-ef
		bandwidth 58
		queue-limit 30
	class class-default
		! WRED
		random-dectect dscp-based
		! WFQ
		fair-queue

interface Ethernet 0/0
	service-policy input voip-be

inerface serial 0/0
	service policy output queue-on-dscp

Low Latency Queueing
LLQ is an option of CBWFQ applied to one or more classes. CBWFQ treats these classes as a strict priority and always services packets in these classes if a packet is waiting. Therefore if you use CBWFQ and use the priority command you have enabled LLQ. This overcomes the biggest drawback of CBWFQ, having a packets with a lower SN but with less latency sensitivity being sent. With LLQ, priority queues are serviced first while guaranteeing bandwidth for traffic in other queues.

LLQ actually polices the priority queue based on the configured bandwidth. The packets in the PQ still have low latency, but LLQ prevents that queue from consuming more than its configured amount. The policing functions of LLQ takes care f protecting the other queues from the LLQ, discarding packets when needed.

LLQ Configuration
Configuration of LLQ is similar to that of CBWFQ, but instead of using the bandwidth command, use the priority command. The priority command sets the guaranteed minimum bandwidth as well as the maximum bandwidth.

Please note, the example below is based on QoS p. 308, however, I have made considerable changes in my answer. If it is not correct I am to blame, not the authors.

    The criteria:

  • R3’s S0/0 is clocked at 128 kbps and is the output interface.
  • R3’s input interface is Ethernet 1/0.
  • VOIP payload is marked with DSCP EF, and placed in its own queue, using tail drop. This class get 58 kbps and is the LLQ.
  • NetMeeting voice and video Server1 to Client1 is marked with DSCP AF41, and placed in its own queue, using tail drop. It get 22 kbps.
  • Any HTTP traffic with “important” in the URL is marked with AF21 and placed in its own queue. The class get 29 kbps.
  • Any HTTP traffic with “not-so” in the URL is marked with AF23 and placed in its own queue. The class get 8 kbps.
  • All other traffic is marked with DSCP BE and placed in its own queue with WRED and WFQ. This class get the remaining 20 kbps.

You can have multiple low-latency queues in a single policy map, and with multiple LLQs each class is policed at the configured rate. You get more granularity in what you police

! All of this is to classify incoming traffic.
! ip cef is for NBAR
ip cef 
class-map match-all dscp-ef
	match ip dscp ef

class-map match-any dscp-af41
	match ip ftp 16383 16384
	match access-group 101

class-map match-all important
	match protocol http "*important*"

class-map match-all not-so
	match protocol http *not-so*"

policy-map incoming-traffic
	class dscp-ef
		set dscp ef
	class dscp-af41
		set dscp af41
	class important
		set dscp af21
	class not-so
		set dscp af23
	class class-default
		fair-queue
		random-detect dscp-based

policy-map outgoing-traffic
	class dscp-ef
		priority 58
	class dscp-af41
		bandwidth 22
	class important
		bandwidth 29
	class not-so
		bandwidth 8
	class class-default
		! This bandwidth command not needed.  
		bandwidth 20
		random-detect dscp-based
		fair-queue

interface ethernet 1/0
	! Output omitted for brevity.
	ip nbar protocol-discovery
	policy-map input incoming-traffic

interface serial 0/0
	! Output omitted for brevity.
	bandwidth 128
	policy map output outgoing-traffic 
Posted in Routing | Leave a comment

CraigsList Crawler 3000

Update 2010-08-26
I have made changes to the script below as a result of some requests. The output should be easier to read.

I should also point out how to find the categories. The usage example that is output when you run the script with no command line switches is only an example, the script will search any category under the “for sale” heading of CraigsList. For instance, under “for sale” is the category “antiques” and when I click on it the link is below.

http://atlanta.craigslist.org/atq/

The category is “atq” in the URL and that is what you would put to search the “antiques” category with this script. The same construct applies if you would like to search “appliances” or any other category.

 Usage: ./CLCrawler3000.pl category keyword
 Exmaple: ./CLCrawler3000.pl sys "mac+mini" 
 Categories: 
 sys == computers
 tls == tools
 bik == bike
 sad == system admin jobs

So if I wanted look for a Linux system administration job I would type in:

 ./CLCrawler3000.pl sad linux

And if I wanted a an armoire in the antique category I would run the script with:

 ./CLCrawler3000.pl atq armoire

Original Post
The name of this script was given by one of my work mates, Scott, when he started using it to search CraigsList. I wrote this script when I became frustrated with the functionality of CraigsList. I live in a small town and I wanted to search for items on CraigsList, however, I would have to search the larger cities around me in order to find items I needed. It didn’t matter to me whether I went to Atlanta, Birmingham or Huntsville, I was still going to have to drive, and when you are looking for bikes on CraigsList you might as well search all of Colorado, California and Texas. The script just grew from there.

I will say that CraigsList has changed its’ output format a couple of times since I wrote this script. I also have had to make changes depending upon the category I was searching. Like all scripts on the internet, your mileage may vary but I hope you find this script as useful as I have.

I would also like to apologize for the code listing. I just used the simple code tag because more fancy highlighting did not look very good.

If you download the script and just run it from the command line, it will give you sample usage. It also outputs a file, clcrawler.html, which you can open in your web browser to view the results.

 Usage: ./CLCrawler3000.pl category keyword
 Exmaple: ./CLCrawler3000.pl sys "mac+mini"
 Categories:
 sys == computers
 tls == tools
 bik == bike
 sad == system admin jobs
#!/usr/bin/perl

use strict;
use LWP::Simple;
use HTML::TokeParser;

die " Usage: $0 category keyword\n Exmaple: $0 sys \"mac+mini\" \n Categories: \n sys == computers\n tls == tools\n bik == bike\n sad == system admin jobs\n " unless @ARGV;

# This is the category
my $cat = $ARGV[0] || "tls";

# This is the keyword you are looking for...
my $keyword =  $ARGV[1] || "surface+plate";

# This is the output file.
my $html = "clcrawler3000.html";

# Define the arrays for each state to be passed into craigslist search,
# by defining each state individually I can tailor my searches quicker.
my %states = (
	Alabama => [ qw(auburn bham columbusga huntsville mobile montgomery tuscaloosa) ],
	Florida => [ qw(daytona keys fortlauderdale fortmyers gainesville jacksonville lakeland miami ocala orlando pensacola sarasota spacecoast tallahassee tampa treasure westpalmbeach) ],
	Georgia => [ qw(atlanta columbusga athensga augusta macon savannah valdosta) ],
	Mississippi => [ qw(gulfport hattiesburg jackson northmiss) ],
	Kentucky => [ qw(bgky cincinnati huntington lexington louisville westky) ],
	SouthCarolina => [ qw(charleston columbia greenville hiltonhead myrtlebeach) ],
	Tennessee => [ qw(memphis chattanooga knoxville nashville tricities) ],
	Alaska => [ qw(anchorage) ],
	Arizona => [ qw(flagstaff phoenix prescott tucson yuma) ],
	Arkansas => [ qw(fayar fortsmith jonesboro littlerock memphis texarkana) ],
	California => [ qw(bakersfield chico fresno goldcountry humboldt inlandempire losangeles merced modesto monterey orangecounty palmsprings redding reno sacramento sandiego sfbay slo santabarbara stockton ventura visalia) ],
	Colorado => [ qw(boulder cosprings denver fortcollins pueblo rockies westslope)],
	Connecticut => [ qw(newlondon hartford newhaven nwct) ],
	Delaware => [ qw(delaware) ],
	DC => [ qw(washingtondc) ],
	Hawaii => [ qw(honolulu) ],
	Idaho => [ qw(boise eastidaho pullman spokane) ],
	Illinois => [ qw(bn carbondale chambana chicago peoria quadcities rockford springfield stlouis) ],
	Indiana => [ qw(bloomington evansville fortwayne indianapolis muncie southbend terrahaute tippecanoe chicago) ],
	Iowa => [ qw(ames cedarrapids desmoines dubuque iowacity omaha quadcities siouxcity) ],
	Kansas => [ qw(kansascity lawrence ksu topeka wichita) ],
	Louisiana => [ qw(batonrouge lafayette lakecharles neworleans shreveport) ],
	Maine => [ qw(maine) ],
	Maryland => [ qw(baltimore easternshore westmd) ],
	Massachusetts => [ qw(boston capecod southcoast westernmass worcester) ],
	Michigan => [ qw(annarbor centralmich detroit flint grandrapids jxn kalamazoo lansing nmi saginaw southbend up) ],
	Minnesota => [ qw(duluth fargo mankato minneapolis rmn stcloud) ],
	Missouri => [ qw(columbiamo joplin kansascity springfield stlouis) ],
	Montana => [ qw(montana) ],
	Nebraska => [ qw(grandisland lincoln omaha siouxcity) ],
	Nevada => [ qw(lasvegas reno)],
	NewHampshire => [ qw(nh) ],
	NewJersey => [ qw(cnj newjersey southjersey) ],
	NewMexico => [ qw(albuquerque lascruces roswell santafe) ],
	NewYork => [ qw(albany binghamton buffalo catskills chautauqua elmira hudsonvalley ithaca longisland newyork plattsburgh rochester syracuse utica watertown) ],
	NorthCarolina => [ qw(asheville boone charlotte eastnc fayetteville greensboro outerbanks raleigh wilmington winstonsalem) ],
	NorthDakota => [ qw(fargo nd) ],
	Ohio => [ qw(akroncanton athensohio cincinnati cleveland columbus dayton huntington limaohio mansfield parkersburg toledo wheeling youngstown) ],
	Oklahoma => [ qw(fortsmith lawton oklahomacity stillwater tulsa) ],
	Oregon => [ qw(bend corvallis eastoregon eugene medford oregoncoast portland salem) ],
	Pennsylvania => [ qw(altoona erie harrisburg lancaster allentown philadelphia pittsburgh poconos reading scranton pennstate york) ],
	RhodeIsland => [ qw(providence) ],
	SouthDakota => [ qw(sd) ],
	Texas => [ qw(dallas houston sanantonio austin beaumont brownsville) ],
	Utah => [ qw(logan ogden provo saltlakecity stgeorge) ],
	Vermont => [ qw(burlington) ],
	Virginia => [ qw(blacksburg charlottesville danville norfolk harrisonburg lynchburg richmond roanoke) ],
	Washington => [ qw(bellingham kpr pullman seattle spokane wenatchee yakima) ],
	WestVirginia => [ qw(charlestonwv huntington martinsburg morgantown parkersburg wheeling) ],
	Wisconsin => [ qw(appleton duluth eauclaire greenbay lacrosse madison milwaukee) ],
	Wyoming => [ qw(wyoming) ],
);

sub get_craigs {

	my $city = shift;

	# Download the page using get();.
	# my $content = get( "http://$city.craigslist.org/search/tls?query=$keyword" ) or die $!;
	print "city == $city\n";
	print "keyword == $keyword\n";
	print "category == $cat\n";
	print "http://$city.craigslist.org/search/$cat?query=$keyword \n";

	my $content = get( "http://$city.craigslist.org/search/$cat?query=$keyword" ) or die $!;

	# Split up the page blob into lines so that we can manipulate them.
	my @lines = split(/\n/, $content);

	foreach my $i (0 .. @lines)
	{
		# This is the key to the whole program, the returned listings are in rows
		# This is the item listing.
		# I tested this on bikes.
#                <p class="row">
#                        <span class="ih" id="images:3n63o53l45O25V35W4a8q669e2752037a111f.jpg">&nbsp;</span>
#                         Aug 26 - <a href="http://auburn.craigslist.org/bik/1920996795.html">Gary Fisher Mountain Bike  -</a>
#                         $950<font size="-1"> (Auburn, AL)</font> <span class="p"> pic</span><br class="c">
#                </p>
		if ((@lines[$i] =~ /href/) && (@lines[$i] =~ /$city/))
		{ 
			print "line == @lines[$i]\n"; 
			my $line = @lines[$i]; 
			print HTML "$line<br>\n";
		}
	}


}

#------------------------------------------------------------------------------
# This didn't really have to be a subroutine, just cleaning things up and making
# them modular.  Open the file.
#------------------------------------------------------------------------------
sub open_html_file {
        open (HTML,">$html")
        or die "Error: cant't open $html \n $!";
}

#------------------------------------------------------------------------------
# Close the file.
#------------------------------------------------------------------------------
sub close_html_file {
        close HTML or die "Error: can't close $html\n $!";
}


#------------------------------------------------------------------------------
# Main.
#------------------------------------------------------------------------------

open_html_file();

# Make html the header
print HTML "<html>\n <head>\n <titel>CraigsList Crawler 3000</title>\n </head>\n <body>\n <br>\n\n" ;

# Iterate through the hash of arrays
foreach my $key ( keys %states ) 
{
	print HTML "<br>$key<br>\n";
	foreach my $i ( 0 .. $#{ $states{$key} } ) 
	{
		print HTML"<br>$states{$key}[$i]<br>\n";
		get_craigs($states{$key}[$i]);
		sleep(5);
	}
        print "\n";
}


print HTML " </body>\n\n" ;
close_html_file();

Posted in Uncategorized | Leave a comment

Ubuntu Join Active Directory

I wrote similar documentation for work and decided to make it generic enough for others to do this. For all I know there is now some GUI tool that does this, however, I believe it’s best to know what’s going on in the background in order to troubleshoot when something goes wrong.

NTP
First set up ntp, because AD, really kerberos, relies upon exact time and DNS.

# apt-get install ntp

Change this line in /etc/ntp.conf

# You do need to talk to an NTP server or two.
server ntp.circus.com

Restart ntp.

# /etc/init.d/ntp restart

Kerberos
Install Kerberos stuff, these will bring some dependencies with them.

# apt-get install krb5-config krb5-user libpam-krb5 libkadm55

Now for a DNS trivia example. Just for fun run this command:

$ dig -t SRV _kerberos._tcp.circus.com

;; ANSWER SECTION:
_kerberos._tcp.circus.com. 600  IN      SRV     0 100 88 a2.CIRCUS.COM.
_kerberos._tcp.circus.com. 600  IN      SRV     0 100 88 a1.CIRCUS.COM.

Now edit the /etc/krb5.conf file:

[logging]
default = FILE:/var/log/krb5libs.log
kdc = FILE:/var/log/krb5kdc.log
admin_server = FILE:/var/log/kadmind.log

[libdefaults]
default_realm = CIRCUS.COM
dns_lookup_realm = true
dns_lookup_kdc = true
ticket_lifetime = 24h
forwardable = yes

[realms]
AD = {
kdc = ad1.circus.com:88
admin_server = ad1.circus.com:749
default_domain = circus.com
}

CIRCUS.COM = {
kdc = ad2.circus.com
kdc = ad1.circus.com
}

[domain_realm]
circus.com = CIRCUS.COM
.circus.com = CIRCUS.COM
[appdefaults]
pam = {
debug = false
ticket_lifetime = 36000
renew_lifetime = 36000
forwardable = true
krb4_convert = false
}

Now test the kerberos install, notice CIRCUS.COM is ALL CAPS:

# kinit judson.bishop@CIRCUS.COM
Password for judson.bishop@CIRCUS.COM:
# klist
Ticket cache: FILE:/tmp/krb5cc_0
Default principal: judson.bishop@CIRCUS.COM

Valid starting     Expires            Service principal
03/04/09 10:29:55  03/04/09 20:30:00  krbtgt/CIRCUS.COM@CIRCUS.COM
renew until 03/05/09 10:29:55

So kerberos works.

Samba
And the samba configuration file:

# cat /etc/samba/smb.conf | grep -v -e "\#" -e "\;" | sed /^$/d
[global]
   workgroup = AD
   password server = ad1.circus.com
   realm = AD.CIRCUS.COM
   security = ads
   idmap uid = 16777216-33554431
   idmap gid = 16777216-33554431
   template shell = /bin/bash
   winbind use default domain = true
   winbind offline logon = false
   winbind separator = +
   winbind enum users = yes
   winbind enum groups = yes
   obey pam restrictions = yes
   allow trusted domains = no
   server string = Hostname %v
   log file = /var/log/samba/%m.log
   realm = AD.CIRCUS.COM
   local master = no
   load printers = yes
   cups options = raw
[homes]
	comment = Home Directories
	browseable = no
	writable = yes
	valid users = %S
[printers]
	comment = All Printers
	path = /var/spool/samba
	browseable = no
	guest ok = no
	writable = no
	printable = yes

Join AD
Please notice the ALL CAPS domain. This is to match you kerberos realm.

# net ads join -U judson.bishop@CIRCUS.COM

You may have to set up you /etc/samba/smb.conf file first. Also, if you have trouble here you may have to go into active directory and delete this computer, then join it to the domain again. The delete and add may require a new kinit process as well. Sorry I can’t be of more help, but this tip may save you a considerable amount of time.

Winbind
Install winbind.

# apt-get install winbind
# /etc/init.d/winbind start

Pam
You need to change all of the “common” files in the directory /etc/pam.d:
Go to that directory and then make a backup just in case:

# for I in `ls common*`; do cp $I $I.sav; done

These are files as they are on my test server, the cat just leaves out the comments because some can be verbose.

/etc/pam.d # cat common-auth | grep -v \#
auth    sufficient      pam_krb5.so ccache=/tmp/krb5cc_%u debug
auth    sufficient      pam_winbind.so use_first_pass
auth    requisite       pam_unix.so nullok_secure
auth    optional        pam_smbpass.so migrate missingok
/etc/pam.d # cat common-password | grep -v \#
password   requisite   pam_unix.so nullok obscure md5
password   optional   pam_smbpass.so nullok use_authtok use_first_pass missingok
/etc/pam.d # cat common-session | grep -v \#
session required        pam_unix.so
session required        pam_mkhomedir.so skel=/etc/skel/ umask=0022


NSSwitch

Edit your nsswitch configuration file, /etc/nsswich.conf.
The following are just the lines you need to change not the whole file.

/etc/pam.d# cat /etc/nsswitch.conf | grep -v \#
passwd:     files winbind
shadow:     files winbind
group:      files winbind


Testing

# getent passwd judson.bishop
# getent group
# wbinfo -t
# wbinfo -u
# wbinfo -g

<code<#net ads testjoin[/code]

#net ads info

From another server, log into Ubuntu using ssh.

ssh -l judson.bishop server.circus.com

More troubleshooting, if you are having trouble with a user:

# id CIRCUS+judson.bishop
uid=16778342(judson.bishop) 
gid=16777729(domain users) 
...output removed...
# ls -ldn /home/AD/judson.bishop/
drwxr-xr-x 3 16778342 16777729 4096 Nov 25 15:56 /home/CIRCUS/judson.bishop/

Playing around with acls:

 cd /home/AD/e09049/
 mkdir Test
 getfacl -R Test >getfacl-Test.txt
 chmod u+rwx Test/
 chmod g+rwx Test/
 chmod o-rwx Test/
 ls -al
 chmod g+s Test/
 setfacl -d -m mask:007 Test
 chown root:AD+854 Test

And finally to play around with group access:

# chown -R judson.bishop:smbgroup /opt/mr/
# chmod -R ug+rwx,o-rwx /opt/mr/
Posted in Uncategorized | Leave a comment