Category Archives: Uncategorized

Ideas?

5
Filed under Uncategorized
Well now. I’ve reached an interesting point. I have run out of ideas for how-to’s. In the past few months I’ve done a lot of work, but it’s all either been stuff that I most certainly don’t want to post online for security reasons (internal infrastructure information) or I sit down to create something, and I think it’s stupid.

I’d say I’m a fairly experienced Linux administrator, with a passion for security and scripting. I don’t know what people want to learn, but I love teaching and writing how-to’s about everything Linux (and especially security!). I figure I might as well write about what people want to learn, so here’s your chance!
Please help Reddit! Just post a comment and I’ll see what I can do :)

"Seems logical to me captain."

Server 2008 – full of sadness and dilusion

0
Filed under Uncategorized

Alright. I’ll admit it first and foremost. I like Windows. Kind of. As I type I’m actually using windows 7, and whatever. I like it. My ATI graphics card isn’t acting like a retarded 4 year old, and I can watch youtube videos at full-screen without issues. Either way, that’s not why I’m angry.
I”m mad with Windows Server 2008. It’s true, it has a lot of easy stuff, and cool features like auditing (also known as PROPER LOGGING…finally), but it also has very retarded things about it. Take Server 2008 Standard edition for example. They restrict your maximum memory to 4GB – because they can. It in no way related to the fact that they can’t address above that amount of memory with 32 bits. They’ve developed PAE (physical address extention) so the kernel natively supports up to 64 GB. In case you don’t understand here’s how it works.

Windows (32 bit version) can only make calls to 2^32 bits of memory

2^32 = 4,294,967,296 bits OR 4.29 GB

With PAE instead of using 32 bits, they extend it to 36 bits

2^36 = 68,719,476,736

The numbers don’t add up exactly, but that’s because of things like overhead and whatnot.

If there’s one thing that pisses me off more than anything else in the IT industry it’s the trickle down effect. Limiting hardware with nothing more than software that refuses to use methods developed to make you pay another 100$ per license makes sense from a business standpoint, but it’s just greasy.

Another thing that really bugs me is the “server core” installs you can do. Having learned about them before I actually got to see what it was like was a huge disappointment. I thought microsoft had finally realized how much more flexible their operating systems can be if they would just get rid of their graphics all together. It’s not for everyone, sure. But don’t piss on my head and tell me its raining. If they REALLY wanted to actually reduce the load on their servers they would actually have a ‘headless’ mode. What the hell is the point in loading a graphical window system into memory to display a command prompt? Hundreds of dlls with unspecified functionality. Hundreds of calls made to your hard drive, scores of kernel modules loaded just to display something that people aren’t even going to be looking at!

It goes along with the Microsoft’s solution. Instead of actually fixing what’s fucking broken, they come out with patches. Instead of fixing their shitty code, or asking for help, they develop DEP. Good job, how’s that working out for you?

It’s not like you can STILL programmaticly disable it. Oh wait, that’s right YOU CAN. Obviously nothing’s perfect and combined with ASLR (Address space layout randomization) it makes it much more difficult, but as long as we’re in the 32 bit world there simply aren’t enough bits to sufficiently randomize memory locations. It just won’t happen for a while.

Keep in mind though this is a vast improvement over server 2003, but show me a windows network that I can’t maliciously become domain admin within a day (a VERY liberal time estimation), and I’ll show you a heard of buffalo that fly.

Just because you replace the shell with CMD.EXE doesn’t mean that you’ve streamlined crap.

Good job M$, that'll totally help.

Linux Forensics & Apple Hate

0
Filed under Uncategorized

At work I’ve *finally* completed documentation for a huge project I’ve been working on. A professor/linux nerd at Tech built a very impressive communication system to get weather and monitoring information from Pico Island, off the coast of Portugal:

http://bit.ly/9IA9z9

Long story short, he passed away. And left NO documentation.

That’s where I came in.

This system was quite impressively put together, unfortunately this professor took over someone’s work machine to do it. Keep in mind this project has being built and added on for the past 10 years or so (at least the data & scripts), so this thing needs 100% (or as close as possible) uptime, because it’s now considered ‘mission critical’.

Did I mention that if it *does* loose power, and restart it gets stuck on boot? Yep. A MISSION CRITICAL SERVER can’t boot up by itself. You have to drop into a shell, disable SELinux, kill the line printer daemon, and restart selinux. Jesus.

My job was to go into this house of cards, and extract the functionality of the server parts, and keep the desktop parts in-tact. I also was told not to access anything unless absolutely necessary. I had to be able to justify every single command I ran if anyone saw it. Financial information, passwords, and other sensitive information that are ‘for eyes only’ were stuff I had to avoid like the plague. It was definitely a trip.

It’s been a long, ridiculously complicated process, but I finally managed to finish tracing every script to environment variables set, files it interacted with, users affected, and research shares used.

I’ve also been given the task of re-installing osx on a macbook (yawn) and transfer settings from one -> another. Easy, right. Wrong. First of all I wanted to make an image. I know, I’ll just flip it over and….what!? There’s no hard drive cover!? Oh well, how hard can it be to get a hard drive out? Well 2 hours, 4 screwdrivers, 24 screws, 2 home-made tools, and 2 frustrated linux nerds later, me and my co-worker Greg finally managed to crack this thing open. Compare that to the 30 seconds it takes me to put my computer into standby, rummage through my drawer to find my screwdriver set, and remove my hard-drive panel. What the hell is wrong with Apple?

So I got it out and used good ol DD to image it (I don’t know any good HFS cloners), then proceeded to re-assemble it. Then I had to transfer the files. Easy enough, right? Hold down T while booting to go into target disk mode (which makes no god damn sense to begin with. Targeting WHAT?). Then use the migration app or whatever to suck all the config and files off. Easy, right? WRONG. “No mac OSX drives found attached to this computer”. Sweet. Thanks again Apple. On a whim (which I knew wouldn’t work) I hit CTRL + ALT + F1 to see if the apple gods would be merciful and give me a message shell. No dice. Of course that wouldn’t work. Why would Apple want to disclose any USABLE information. That’s like going to a murder investigation and just repetitively screming “SOMEONE DIED. SOMEONE DIED. SOMEONE DIED.” who the hell does that help?

The next logical step was to just gut the machine again (which I did), so another   24 screws later I hooked it up and sucked the stuff off. If I was able to diagnose why the target mode wasn’t working in the first place I would’ve saved myself lots of time, frustration, and urges to smash these laptops into a billion little pieces.

Long story short – fuck you Apple. I’ll buy your embedded devices, but your computers are over-priced pieces of shit.

Don’t worry, I’m not dead. It was just my free time ^_^

1
Filed under Uncategorized

Hi all,

So I got the job over at EIT – Systems as a Linux administrator (it’s pretty much me and my co-worker Greg, as well as some west-ites). I do a lot of scripting and Linux administration which I enjoy a whole ton more then user support. I mean don’t get me wrong, I don’t mind helping people. Or even dealing with people, it was just getting old helping people fix printers day in and day out.I’m actually challenged where I am now, and I many times feel like a mad scientist conjuring up the fruits of his understanding. I much prefer where I am right now, because I work with 3 other people, and 90% they’re not there (they work mornings). So since I’m working from 2-6 everyday I work mostly by myself.

They’ve also given me my own workstation, so I’m not forced to use RHEL 5, I can use my distro of choice – Ubuntu. I also have ROOT ACCESS to my computer, so I can install whatever I please, so I now have a bitchin setup with copiz and tilda. Buttons F7-F12 are assigned to tilda screens, so if I want to check my mail log (because I have an SMTP relay firewalled off, with a WHOLE ‘nother post about my custom port knocker) I simply press the F8 button. Firewall rules in place right now? F9. /var/log/messages? F11 :) . F12 is assigned to a pop-up terminal (the MOST handy thing I’ve ever done to any distro). I used to use yakuake, because that was the first terminal emulator of the sort I was exposed to, but tilda is gnome-native so I don’t have to download a bunch of KDE packages to make everything play nice. I love my dual monitor setup with virtual desktops + compiz. Today I had an entire lab’s worth of RDP sessions open and was just spinning my cube around to each of them, and had my ‘main desktop’ so I could just switch to that right quick when I needed to!

Just yesterday actually I *finally* got root access! I’ve had sudo access to things that I’ve been able to elevate to a root shell, but now I’ve got it all :) . It was really the first time I’ve needed it (and there was no other way to really avoid it), because everything else could be done on my behalf. A bit of a pain in the ass though if I’m working when no-one’s around and I need to do something. I’m glad that I have worked myself into a position where I can be so heavily trusted with the resources at Michigan Tech. It feels good :) .

So I’ve also been REAL busy lately working on a side project for EIT – Systems, and I’m trying to keep it as deep under wraps as possible. This is the only thing I’ll give for any of my co-workers that happen to be reading.

Good ol' Grub4DOS - a bit of hex magic to get the header though :)

There was a bit of a deficit with working on it though. My roommate’s laptop decided to melt his graphics card, so I let him use my ‘test bed’ laptop that I use for the development of the EIT Diagnostic Toolkit. Thankfully though after spring break he brought a desktop up, so I’m back to developing!

I also turned 21 on the 7th! It’s nice not having to deal with playing hey mister :-P . It feels as though I’ve finally made the transition (at least a little bit) into the ‘bubble’ of society.

I’ve also been playing a lot of guitar (it’s pretty much back to work, guitar, school, or development), and that has pretty much filled up all my social time as well. I wish that I still wanted to go out, but working on this project is so rewarding, I’m doing something constructive and have something to be proud of at the end of the day. That and I have a house that I can drink beer in instead ^_^.

I’m also going to try something new. I’m going to try to enclose a bit of Linux knowledge with every post, so I give you – My PS1!

\n\[\e[33;1m\]#(\[\e[31;1m\]`if [ $? = 0 ]; then echo ‘\[\e[32;1m\]^_^\[\e[0m\]‘; else echo ‘\[\e[31;1m\]O_O\[\e[0m\]‘; fi`\[\e[33;1m\])-(\[\e[34;1m\]!\!\[\e[33;1m\])-(\[\e[34;1m\]\u@\H\[\e[33;1m\])(\[\e[34;1m\]\T\[\e[33;1m\])-(\[\e[34;1m\]\[\e[33;1m\])\n#\[\e[1;32m\](\w)\[\e[33;1m\]\n\[\e[0;32m\]

Now I know it looks a bit daunting, but when I wrote it I had the ASCII colors mapped to variables (done in my .bashrc) so I could do it.

Here is my .bashrc (a bit weird because we use C shell by default, so my .login just spawns a bash shell):

txtblk=’\e[30;1m'  # Black - Regulan
txtred='\e[31;1m'  # Red
txtgrn='\e[32;1m'  # Green
txtylw='\e[33;1m'  # Yellow
txtblu='\e[34;1m'  # Blue
txtpur='\e[35;1m'  # Purple
txtcyn='\e[36;1m'  # Cyan
txtwht='\e[33;1m'  # White
bldblk='\e[1;30m'  # Black - Bold
bldred='\e[1;31m'  # Red
bldgrn='\e[1;32m'  # Green
bldylw='\e[1;33m'  # Yellow
bldblu='\e[1;34m'  # Blue
bldpur='\e[1;35m'  # Purple
bldcyn='\e[1;36m'  # Cyan
bldwht='\e[1;37m'  # White
unkblk='\e[4;30m'  # Black - Underline
undred='\e[4;31m'  # Red
undgrn='\e[4;32m'  # Green
undylw='\e[4;33m'  # Yellow
undblu='\e[4;34m'  # Blue
undpur='\e[4;35m'  # Purple
undcyn='\e[4;36m'  # Cyan
undwht='\e[4;37m'  # White
bakblk='\e[40m'    # Black - Background
bakred='\e[41m'    # Red
badgrn='\e[42m'    # Green
bakylw='\e[43m'    # Yellow
bakblu='\e[44m'    # Blue
bakpur='\e[45m'    # Purple
bakcyn='\e[46m'    # Cyan
bakwht='\e[47m'    # White
txtrst='\e[0m'     # Text Reset
txtgrn2='\e[0;32m' # Other Green
IP=`curl -s www.whatismyip.org`
PS1="\n\[$txtwht\]#(\[$txtred\]\`if [ \$? = 0 ]; then echo ‘\[$txtgrn\]^_^\[\e[0m\]‘; else echo ‘\[$txtred\]O_O\[\e[0m\]‘; fi\`\[$txtwht\])-(\[$txtblu\]!\!\[$txtwht\])-(\[$txtblu\]\u@\H\[$txtwht\])(\[$txtblu\]\T\[$txtwht\])-(\[$txtblu\]$IP\[$txtylw\])\n#\[$bldgrn\](\w)\[$txtwht\]\n\[$txtgrn2\]”

Now my PS1 is a bit more readable (only if you really know what you’re looking at though).

This is what it looks like:

Before we pull the PS1 apart, lets be clear about what is going on. Firstly is the #’s, they seem just for show, but when I’m scripting it’s a lot easier for me to copy entire blocks, and just use sed to remove all the lines that start with a #, work smart. Not hard.

The next section is a graphical representation of the last program’s exit status. I didn’t really realize how much of an impact this would have on my scripting but now I use exit codes a LOT more, which allows for greater flexibility.  ^_^ = True (exit 0) – 0_0 = False (exit !0)

Next we have the command number I’m on. This is handy because if I ever want to repeat a command, instead of selecting the entire thing, I can just type in !XXX where XXX is the command number, and it will look it up in the history and execute!

After that we have (user@hostname) & (time) – self explanatory

Then with a bit of cURL magic we grab our external IP address:

IP=`curl -s www.whatismyip.org` – This is what i’ve always used but they seem down, this next line works for now

IP=`curl -s whatismyip.com/automation/n09230945NL.asp`

On the next line we just have the working directory, which makes life a LOT easier when trying to figure out where the hell you are! Especially if you have as many terminals open as I do on a daily basis!

Alright, so now that we know what our PS1 is doing, we can try to understand it.

The first relevant part is this:

`if [ \$? = 0 ]; then echo ‘\[$txtgrn\]^_^\[\e[0m\]‘; else echo ‘\[$txtred\]O_O\[\e[0m\]‘; fi\`

This is a logic test to check the $? variable (the exit status of the last program) and selects either the ^_^ or the 0_0

The rest of it is pretty much just built-in’s

\[$txtwht\])-(\[$txtblu\]!\!\[$txtwht\])-(\[$txtblu\]\u@\H\[$txtwht\])(\[$txtblu\]\T\[$txtwht\])-(\[$txtblu\]$IP\[$txtylw\])\n#\[$bldgrn\](\w)\[$txtwht\]\n\[$txtgrn2\]”

Here’s a list of built-ins:

  • \a : an ASCII bell character (07)
  • \d : the date in “Weekday Month Date” format (e.g., “Tue May 26″)
  • \D{format} : the format is passed to strftime(3) and the result is inserted into the prompt string; an empty format results in a locale-specific time representation. The braces are required
  • \e : an ASCII escape character (033)
  • \h : the hostname up to the first ‘.’
  • \H : the hostname
  • \j : the number of jobs currently managed by the shell
  • \l : the basename of the shell’s terminal device name
  • \n : newline
  • \r : carriage return
  • \s : the name of the shell, the basename of $0 (the portion following the final slash)
  • \t : the current time in 24-hour HH:MM:SS format
  • \T : the current time in 12-hour HH:MM:SS format
  • \@ : the current time in 12-hour am/pm format
  • \A : the current time in 24-hour HH:MM format
  • \u : the username of the current user
  • \v : the version of bash (e.g., 2.00)
  • \V : the release of bash, version + patch level (e.g., 2.00.0)
  • \w : the current working directory, with $HOME abbreviated with a tilde
  • \W : the basename of the current working directory, with $HOME abbreviated with a tilde
  • \! : the history number of this command
  • \# : the command number of this command
  • \$ : if the effective UID is 0, a #, otherwise a $
  • \nnn : the character corresponding to the octal number nnn
  • \\ : a backslash
  • \[ : begin a sequence of non-printing characters, which could be used to embed a terminal control sequence into the prompt
  • \] : end a sequence of non-printing characters

So that’s that. I promise to try to keep this up though, because I may finally have some time to put aside!

Stay thirsty!

Long time, no post

0
Filed under Uncategorized

Hi all,

Sorry for the delay in posts, I’ve been uber busy with thanksgiving break and all that, and the only internet I had was my iPhone tether. I realize that’s quite enough bandwidth but I was busy with a certian someone for the rest of the week, and dealing with it blowing up in my face (freaking emotional baggage!). Either way, time to dive face-first into learning and working (and posting). I need to get out of my own head anyway, and nothing does that like being 500 miles from anyone in the deep north. Blah.

I’ll be posting more tonight probably, assuming inspiration hits me, if not…::shrugs:: I dunno what to tell you

Stay thirsty friends.

Virtualbox woes :(

2
Filed under Uncategorized

So I got fed up with using the Windows 7 release candidate, and I decided to do a fresh install, so I backed up everything that I thought I’d need (INCLUDING my virtualbox drive + snapshots), and did the upgrade.

I’ve been working on (and documenting, thank god) getting the SunRay server software installed in ubuntu and passing X-sessions to the stateless SunRay 1. I was going to install LTS so I could compare the difference in performance, and ease of use. I just started the virtual machine, and lo and behold everything is super-broken. My fears are confirmed, and I DO have to re-install everything. Hopefully I can script the majority of it, but either way it’s still a bunch of time wasted :( .

I’m also installing RHEL5 in another VM to test-install some software (and probably script it out so when I DO have to deploy it I can just do it in a couple steps.

The rule of thumb? Don’t expect things to ‘just work’ because installing a new operating system can do quite a bit of damage to something like a vm.

Crap.

sun-logo-new

Stupid logic joke. I know, and i’m sorry :-P

I can haz promotion?

0
Filed under Uncategorized

So today I got an email from one of my bosses (Thomas Gemignani) about ‘wanting to meet’. Of course my brain over thinks these kind of things a lot, so I was mulling over what I could have possibly done wrong, and stressing about it all day. I was only concerned because of the fact that I JUST had a review with one of my USM (user support managers). This was his boss that wanted to talk to me :-/.

I walked into the EECN (East end computing network) office, and he was already talking to someone. The little bit that I DID overhear made me realize that this wasn’t anything even remotely negative, just another performance review. So I sat down and buried myself in my 13 page paper on getting SunRay Server Software working properly (proofreading == suck).

The time finally did come for me to talk to him, and since I don’t work directly under him, we both agreed that it was a bit hard to truly evaluate my performance. He is a very easy going guy, so it was really easy to talk to him about every thing that I’ve done, good and bad, working for MTU.

It was at that point where he brought up my technical skills, and how he feels like he’s “under-utilizing me”. I was a bit hesitant to agree, as getting this job in the first place was a HUGE opportunity for me. I don’t typically push, and I’m thankful for what I DO have.  Although now that I’ve felt out the MTU network(s), I can be a lot more productive in a higher up position. I hate rocking the boat, asking for more money and all that jazz, so I’m glad that I didn’t have to push for it. I have always been taught to be thankful for what I’ve got, but the ability to play hardball sometimes benefits a lot. Tom was pretty much the only person that I felt comfortable talking about a promotion of any kind to, not to say that my other bosses aren’t easy going, he just thinks like I do.

That’s not to say that I wasn’t a bit hesitant to ask about a possible pay increase, but I mean what’s the worst that could really happen?

He then proceeded to tell me that he would talk to the other admins, and fight tooth and nail for me (if need be) to get bumped to a systems position, and pay raise if it was even remotely possible.

I’ve got a new HowTo coming on getting Sunray thin-clients working with ubuntu (audio and USB support included). I used a tutorial I found on the Sun-ray.org website, but I spiced it up a bit, and I’m working on getting another better smartcard system going, but thats down the line a little bit.

Its a good day :)

Movin on up!

No more free hosting!

0
Filed under Uncategorized

Hi all,

Seeing as I have taken quite an interest in this, I have decided to forget about just alright free hosting on wordpress.com, and I have moved to a full on host :)

The new website is:

http://talesofacoldadmin.com/

Not too much different, but a LOT more stuff to play with for me :)

I hope you guys continue to read, and enjoy!

Its very nice, I like very much

“My Cluster”…Making RAM faster with every simulation (hopefully).

0
Filed under Uncategorized

I use both the words “My” and “Cluster” in quite a loose sense. Mainly because it’s not mine, and it’s not quite a cluster. It is owned by Brian Davis, one of my professors here at tech, but administrated by me, funded by grant money for a project that he got a few years ago.

Essentially the grant (and simulations) are to find a better way to organize memory. If you think about the way that memory is in the conventional sense, you have programs interacting with first the CPU cache, then to the RAM, then to the HD as a ‘last resort’. Got it. Right.

What he’s aiming to do is essentially provide another layer between the RAM and the HD (and in the future between the RAM and the CPU cache) that essentially has an algorithm to organize it, both physically (reducing copper wire latency) and programatically (in page size, as well as frequency of use). I’m not too sure on the specifics past that, and that was only one branch of what he’s actually researching, but it struck me as quite interesting!

I’m hoping at some point to get job management going properly, as well as “actual clustering” going. Right now people just ssh into the head, and have a trust setup between head and nodes, so they don’t have to login after they’ve authenticated through the head. The nodes are pretty easy to start and compile code on, so it works for all the simulations, at least for now.

Every node is running Redhat Enterprise Linux 5, and the simulations are written in C. I’ve got everything setup to kickstart whenever I need it. Take tonight for example, my nodes have been mysteriously dying, and I have a feeling that someone’s code corrupted something in the T3 module, and they just kept trying to run it on all the nodes.

Thankfully I have set it up so its nice and easy to wipe everything clean. Thank god for kickstart, this is a copy of mine :

http://pastebin.com/f6c745c08

Running through my kickstart, you see that it puts every node on the same NIS domain (btdpool.ee.mtu.edu), and uses an ftp server to grab all the necessary packages and files. No X windows, as there isn’t a need for it. Generates the swap partition depending on memory size, and fills the rest of the partition with ext3 (forget LVM!). Installs all the packages it really needs, and on the post install it does the following:

Changes the passwd binary for all nodes to yppasswd, so when people try to change their password, it propagates properly through the NIS domain.

Adds “+::::::” to /etc/passwd – this basically states that the rest of the file should be looked up in “Yellow pages” (or the NIS server, whichever you prefer :-P )

Adds “+:::” to /etc/group, and does the same exact thing as the previous one, but with groups instead of passwords.

Puts an entry in /etc/fstab to connect to the NFS share I have running for everyone’s homedrive, allowing SSH trusts to remain in place no matter how many times I wipe them :) .

These things seem relatively simple…now. Trust me, it took me many many hours to figure out exactly what I needed to do to get everything working like this. Now all I have to do to “refresh” the nodes is PXE boot them, and reboot after they’re done. Since nagios only checks SSH and ICMP connectivity to all nodes, it will still be fully functional.

All in all, it has been a huge learning experience for me, getting to know a lot about RHEL administration. It’s been a lot of fun, and I’ll get into all the scripting that I did to make everything work flawlessly :) .

I’ve got a lot of things happening using cron and bash scripting, but I don’t feel like getting into that right now.

Also look at the blast from the past that I found today at work!

Top

Right

Front

Left

Back

I saw it sitting there and literally burst out laughing. There was a huge cart of old crap sitting on the left when I walked in and I spent about 20 min digging through it for nostalgic things like this :) .

LOPSA

0
Filed under Uncategorized

Horay! I just joined the League of Professional System Administrators.I have a feeling this is going to be a good outlet for both learning as well as finding contacts in the future. I joined on the advice of Matt Simmons, so thanks for letting me know about it!

If you’re interested in System Administration, you should check it out, its only 25$ to join if you’re a student :) .

http://lopsa.org/

 

Also I joined both of my zeroshell tutorials together into a page, check it out!

http://talesofacoldadmin.wordpress.com/migrating-from-linksys-linux-with-vpn/