Archive for the ‘Linux’ Category

Fixing an LVM in Ubuntu

Monday, November 7th, 2011

WARNING: There are steps, references, commands and/or instructions in this article that can be very dangerous to your filesystem and possibly cause irreversible damage to your data.  You are responsible for maintaining your data – use these commands only after you fully understand the implications of what they do and you’re comfortable and competent.  Any damage resulting from using any of these commands or guidance of this article is fully your responsibility – I can’t be responsible for your data.  By continuing, you agree to release the author, the hosting service for this site, anybody posting comments to this site, your dog, etc. free from liability should any damage occur, whether accidental or intentional.

I use Ubuntu quite a bit and have recently started using LVM on an internal FTP/TFTP/SCP “drop box” of sorts. The reason for LVM is that I needed to dynamically “grow” the amount of disk space available for the server (it was full). Being a VM, it’s easy to carve out another virtual disk and assign it to the VM. LVM takes over from there…

This is all fine-and-dandy if your storage is stable, however we ran into an issue where storage was oversubscribed and completely full on the SAN. This particular server was running on a NetApp SAN and the volumes ended up being taken offline. This happened automatically on the SAN. After freeing up some space for the remaining volumes, I brought the volumes back up. Either this process itself or having me shutdown the VM (I hadn’t noticed that the storage was gone at the point I shut the VM down) resulted in a corrupted filesystem.

Upon trying to boot the server back up, I was presented with a BusyBox screen and several error messages indicating that the filesystem was messed up. I received a message like “Target filesystem doesn’t have /sbin/init” on the screen (with the BusyBox prompt).

Here’s what I did to fix the filesystem and restore the system to full functionality, and a lesson I learned.

To restore the system:

  1. Download the SystemRescueCd ISO (www.sysresccd.org)
  2. Create a new virtual disk (for copying data that I couldn’t live without to) and assign this to the VM
  3. Mount the ISO in the VM on startup and boot off of the CD
  4. Setup the networking (assign an IP address, default gateway, etc.)
  5. SSH into the server
  6. Check the LVM for filesystem errors
    1. e2fsck -n /dev/mapper/<LVM name>
    2. If it shows errors, you might want to continue to fix these errors (if possible)
  7. Mount the old LVM
    1. mkdir /mnt/t (t for temp, or whatever directory name you desire)
    2. mount /dev/mapper/<LVM name> /mnt/t
  8. Create a partition on and format the new virtual disk
    1. Plenty of resources on this – Google for the filesystem you’re wanting to use (ext2, ext3, ext4, etc.)
  9. Mount the virtual disk
    1. mkdir /mnt/n (n for new – again, whatever you want)
    2. mount /dev/sdc1 /mnt/n
  10. Copy data that I couldn’t live without from the old partition to the new one (so from /mnt/t to /mnt/n)
  11. Unmount and fix errors on the old LVM partition
    1. umount /dev/mapper/<LVM name>
    2. e2fsck -v /dev/mapper/<LVM name>
      1. I know there are ways to have e2fsck automatically fix errors, but I wanted to see and approve each error, so I went the somewhat slow path
  12. Shutdown the VM, remove the SystemRescueCD ISO and the new virtual disk
  13. Try booting the VM

After following these steps (best as I can remember), the system was back working again.  In this system, I had a single LVM that was pretty much an “everything” partition – root, FTP/TFTP/SCP storage, etc.  This leads up to my lessons learned:

  • On file servers, particularly VMs, it’s so easy to carve out additional virtual disks, keep your root filesystem (boot loader, kernel, etc.) on one virtual disk that’s only used for base OS functionality.  Create another virtual disk(s) for file storage, using LVMs if necessary.  This allows you to still easily access the data by simply assigning the virtual disk to another “clean” Ubuntu install, bypassing the need to boot off of the SystemRescueCD ISO.  In my instance, everything was combined.  Since it was on a SAN, I had assumed very little risk of data corruption (messed up filesystem), however oversubscription can cause problems.
  • Use oversubscription on your storage sparingly and in a planned fashion.  This bit me, and I’ve heard other IT professionals say “Oh, don’t worry about it – it won’t use it”.  While it might be unforeseeable that a system consume all of the allocated space, it is possible and should be guarded against.  What happens if it does consume all of the space (logs, updates, etc. can all contribute to filesystem growth).  Ensure that your core, mission-critical systems are NOT using oversubscribed storage.
  • When working with LVMs, don’t point to the physical disks (/dev/sdb) and partitions (/dev/sdb1) for troubleshooting – it’ll get you nowhere.  Using the pvdisplay, lvdisplay, etc. commands (for examining your LVMs), you’ll be able to see the LVM name, as well as partitions that comprise the LVMs.  Focus on the LVM, not the partition (at least in my case).  This isn’t to say that sometimes there are physical issues occurring (SAN stats or SMART errors on a local disk should help here).

So, your mileage may vary, but this is what I did to fix the LVM filesystem (and restore functionality of my system).

What do you consider best-practices for Linux VM creation as well as general filesystem tasks?  Do you have a different tip or trick than I’ve mentioned above?

Until next time…

My journey with ioctl() on Linux

Wednesday, October 10th, 2007

Background

Lately, I’ve been working on an application that heavily uses the ioctl() function for different settings with hardware devices (get/set NIC config – IP, subnet, broadcast, MTU, etc.). This function is so versatile and has so many different uses, that it can be difficult at times to understand what exactly is needed for it to work properly. (more…)

Interaction of stdout buffering using standard stdio.h functions when using pthreads

Thursday, September 27th, 2007

I’ve been working on a project recently that stretched my C programming ability (yes, I’ve been learning quite a bit — especially since I’d never really tackled any significant programming on Linux prior to this).  The app is multithreaded, using the pthreads library.

I ran into a situation where I had multiple threads executing simultaneously, but the text output wasn’t being displayed. (more…)

HOWTO dual-boot between Linux and Windows using GRUB

Tuesday, May 8th, 2007

When I first started out with Linux, I wasn’t aware of some of the issues that can arise with dual-booting Windows with another OS.  The following ramblings are some of my experiences/findings. (more…)

Linux Command Reference

Tuesday, May 8th, 2007

Here’s a Linux Command Reference that I created a couple of years ago when I was first learning Linux, to help me with some of the different commands available. Obviously it’s not complete and could use some updating. After all, let’s face it — Linux is always changing — growing, morphing into the great OS it is! Furthermore, depending on the distro you’re using, you may or may not have rpm at your disposal — maybe yum will be the choice, or apt-get the package tool for the distro.

When I created this command “cheat sheet”, I was using a Fedora Core distro. Since then I’ve moved on to an Ubuntu (Debian) based distro. Don’t flame me or rant about one distro over the other here — that’s not why I’m mentioning this. Simply, some commands may only work in a FC-based distro (such as rpm, unless you install a package that handles rpm files).

Here’s the document in PDF format:

Linux Command Reference (common_linux_commands.pdf)

In closing, as I typically do, I welcome feedback. Let me know if you have any sections or additional commands that’d be helpful to add to this!

Installing OpenNMS on Fedora Core 2

Thursday, January 13th, 2005

OBSELENCE NOTE: This was written well over a year ago, so may be outdated (ie. may have broken links — if so, let me know and I’ll update).  Just because it’s old doesn’t mean that it’s not worth sharing!

Summary:

A quick how-to on setting up OpenNMS (and some of the needed libraries) on Fedora Core 2.

Instructions:

For those who aren’t familiar with OpenNMS, it’s a program mean to aid in the monitoring of a network(s) (see www.OpenNMS.org for more info).

Originally, I tried installing OpenNMS on Red Hat 9, but had a lot of issues with older library packages.  Being faced with upgrading a *ton* of libraries, I ultimately decided to install Fedora Core 2 (FC2).

Prerequisites

This how-to will assume the following:

  • Apache is installed
  • DB4 (Berkeley DB) is installed
  • Postgre SQL is installed
  • RPM management tools (the rpm program/script, etc)

When I installed FC2, I wasn’t low on space, so I pretty much told it to install most everything (not all packages from all four CDs — but the most common packages — the ones displayed in the package selection list).

 

Install Required Libraries

The following libraries need to be installed (to install, run: rpm -ivh <package name>):

  • net-snmp-5.1.1-2.i386.rpm (disc 1)
  • net-snmp-perl-5.1.1-2.i386.rpm (disc 3)
  • compat-libstdc++-7.3-2.96.126.i386.rpm (disc 3)
  • postgresql-jdbc-7.4.2-1.i386.rpm (disc 3)

All of the above libraries that have a (disc #) after them can be found directly on the FC2 CD specified.

 

Install J2SDK

To avoid plagarism and redundancy, see http://wiki.opennms.org/tiki-view_faq.php?faqId=1#q16 for where to get the correct J2SDK (along with how to use it).

 

Install Jakarta/Tomcat

Before you begin doing anything:

Step If this fails, try…
Check to make sure Apache is running

Make sure Apache is running:

[user@system src]# /etc/init.d/httpd status

If Apache is running, you’ll see:

httpd (pid # # # #) is running…
[user@system src]#

Otherwise you’ll see:

httpd is stopped…
[user@system src]#

 

If the httpd (Apache) service is running, go in a web browser to http://127.0.0.1/ on the local machine, or go to http://<machine IP> from another machine’s web browser.   You should see the default Apache page.  If you get an error, something’s messed up.  Fix the problem with Apache before continuing!

Check to see if httpd is configured to start with the system

 

No automatic startup:

[user@system src]# chkconfig –list httpd
httpd     0:off 1:off 2:off 3:off 4:off 5:off 6:off

To correct this:

[user@system src]# chkconfig httpd on
[user@system src]#

 

With automatic startup:

[user@system src]# chkconfig –list httpd
httpd     0:off 1:off 2:on 3:on 4:on 5:on 6:off

Follow the instructions found at http://wiki.opennms.org/tiki-view_faq.php?faqId=1#q14.  I found that this is also a good place to place the CATALINA_HOME environment variable.  My /etc/profile.d/java.sh file looks like the following:

[user@system src]# cat /etc/profile.d/java.sh
export JAVA_HOME="/usr/java/j2sdk1.4.0_02/"
export CATALINA_HOME="/var/tomcat4/"
declare -x JAVA_HOME="/usr/java/j2sdk1.4.0_02/"
declare -x CATALINA_HOME="/var/tomcat4/"
[user@system src]#

Note that as of now, there’s nothing in /var/tomcat4/, but we’re getting to that right now!  That’s where the Jakarta/Tomcat program goes — for now, save yourself a step later and put it in there.

Okay… moving on from the above stuff to check before installing Jakarta/Tomcat.  Go to jakarta.apache.org -> Downloads (Binaries link) -> Scroll down in the long list until you reach Tomcat 4.1.30 -> Download jakarta-tomcat-4.1.30.tar.gz (I usually save stuff like this in /usr/src/ — I’m going to assume you’re doing the same).  Go to /usr/src (cd /usr/src).  Run tar -xzvf jakarta-tomcat-4.1.30.tar.gz.  After it finishes expanding the tar:

[user@system user]# cd /usr/src
[user@system src]# tar -xzvf jakarta-tomcat-4.1.30.tar.gz
[user@system src]# mv jakarta-tomcat-4.1.30/ /var/tomcat4/
[user@system src]#

Okay, now you’ve got Tomcat (Jakarta — I’m calling it Tomcat from now on) where the OpenNMS RPMs expect it.  Modify the /var/tomcat4/conf/tomcat-users.xml file to add another user with admin/manager privileges, similar to:
         <user username="admintest" password="test" roles="admin,manager"/>

Let’s startup Tomcat:

[user@system user]# $CATALINA_HOME/bin/startup.sh
Using CATALINA_BASE:   /var/tomcat4
Using CATALINA_HOME:   /var/tomcat4
Using CATALINA_TMPDIR: /var/tomcat4/temp
Using JAVA_HOME:       /usr/java/j2sdk1.4.0_02
[user@system src]#

Go ahead and test it — go to 127.0.0.1:8080 (on the local machine) or <ip address>:8080 on a remote machine.  If you get the Jakarta/Tomcat welcome page, you’re good to go!  If not, go over the above steps again, search the net for the problem, etc.

 

Install OpenNMS

Go to www.OpenNMS.org -> Downloads (under Featured Links heading) -> download all three of the RPMs for Fedora Core 2 (once again to /usr/src/ ?):

  • opennms-1.1.3-2.fc2.i386.rpm
  • opennms-docs-1.1.3-2.fc2.i386.rpm
  • opennms-webapp-1.1.3-2.fc2.i386.rpm

Go to http://people.ee.ethz.ch/~oetiker/webtools/rrdtool/ and get RRDtool (Download -> Stable Release -> scroll down until you find the right RPM):

  • rrdtool-1.0.41-1.8.0.i386.rpm

Now install RRDtool:

[user@system user]# rpm -ivh rrdtool-1.0.41-1.8.0.i386.rpm
<a bunch of stuff will be shown here>
[user@system src]#

Note that if any errors are generated, you need to get them fixed (mainly missing dependecies) before continuing — if you get errors — fix them then retry this step.

Now install OpenNMS:

[user@system user]# rpm -ivh open*.rpm
<a bunch of stuff will be shown here>
[user@system src]#

Note that on my system, even though I had the necessary dependencies (modules) installed, I was still getting "missing package" errors.  To avoid that, I used the following command to install OpenNMS:

[user@system user]# rpm -ivh –force –nodeps open*.rpm
<a bunch of stuff will be shown here>
[user@system src]#

BE VERY CAREFUL USING THE ABOVE COMMAND — IT TURNS OFF ALL DEPENDENCY CHECKING AND WILL FORCE AN INSTALLATION, WHETHER IT WILL OR WILL NOT CAUSE PROBLEMS!!!  (use it only if you’re sure that you’re safe and know what you’re doing)

 

Before starting the service, you need to customize the IP range(s) that the discovery service will search/monitor.  Modify the file /opt/OpenNMS/etc/discovery-configuration.xml to include the IP range(s) that you want OpenNMS to monitor.  Refer to the HTML page at /opt/OpenNMS/docs/quick/book.html for more information on customizing this and other options.

At this point, if the above step didn’t generate any errors, you’re ready to start the OpenNMS service.  To do this:

[user@system user]# /etc/init.d/opennms start
Starting OpenNMS:                      [  OK  ]
[user@system src]#

If you have an error above (it would say FAILED instead of OK), look at the logs found in /var/log/opennms/ to try and see what’s causing the problem.

If the above started without an error, before accessing OpenNMS, make sure that all (or most of) the services are started:

[user@system user]# /etc/init.d/opennms status
OpenNMS.Eventd          : running
OpenNMS.Trapd           : running
OpenNMS.Dhcpd           : starting
OpenNMS.Actiond         : running
<blah, blah, blah — more services listed>
[user@system src]#

If the the services are good to go, in your web browser, go to 127.0.0.1:8080/opennms (local machine) or <ip address>:8080/opennms (remote machine).  The default login is admin/admin.  From there you can create new user accounts, etc.

That’s it!

 

Handy Directories to Know About

/var/lib/pgsql/ = Postgre SQL directory
/var/log/opennms/ = OpenNMS log directory
/usr/java/j2sdk-??? = Java SDK directory
/opt/OpenNMS/ = OpenNMS directory
/opt/OpenNMS/docs/ = OpenNMS documentation directory (HTML format)
/var/tomcat4/ = directory that OpenNMS RPMs looks for Jakarta/Tomcat in (during the setup, modifying the server.xml file)

 

FEEDBACK

Do you have some comments (errors, omissions, etc) about this brief (and somewhat disorderly) how-to?  Let me know!