okgg.org -- View topic - Parse file into another file and modify it.

SmartFeed

okgg.org > Forum Index > All Things Technical > Parse file into another file and modify it.

Author

Thread

TehDanMan
Guerilla Ontologist

Joined: 01 Jul 2003
Posts: 13143

Parse file into another file and modify it.

Each line reads like this:

Code




user1:passwordhash


user2:passwordhash


user3:passwordhash

I need to parse that file into another formatted like this:

Code




user1:user1


user2:user2


user3:user3

So far what I have is this command to do a bit of cleanup, but it doesn't do everything I need. I'm at the limit on my knowledge of sed.

Code




#!/bin/sh


cat /path/to/passwd | sed 's/:[^ ]*/:??/' > /path/to/userlist

That outputs this:

Code




user1:??


user2:??


user3:??

I imagine it would only take a short extension of my one liner shellfoo.
_________________
Blargle Flarg Bergah Merg

Wed Jan 13, 2010 11:17 am

LightningCrash
Smile like Bob, order your free LC today

Joined: 03 Apr 2003
Posts: 5020

Code




lc@desktop:~/tmp$ cat test


user1:passwordhash


user2:passwordhash


user3:passwordhash


lc@desktop:~/tmp$ cat test|awk -F":" '{ print $1":"$1 }'


user1:user1


user2:user2


user3:user3

If you need anything else I will do my best to help.

Wed Jan 13, 2010 11:21 am

TehDanMan
Guerilla Ontologist

Joined: 01 Jul 2003
Posts: 13143

Thanks man, but I got it done with perl and a friend's help.

Code




#!/usr/bin/perl





open(FILE, "<@ARGV[0]");





while(<FILE>){


 my ($user) = (split(':', $_))[0];


 print "$user:$user\n";


}


close(FILE);

_________________
Blargle Flarg Bergah Merg

Wed Jan 13, 2010 11:52 am

LightningCrash
Smile like Bob, order your free LC today

Joined: 03 Apr 2003
Posts: 5020

np.

If you want to read on AWK I highly recommend this book:
http://www.amazon.com/AWK-Programming-Language-Alfred-Aho/dp/020107981X

The authors of the book are the guys who wrote AWK.

Wed Jan 13, 2010 3:14 pm

TehDanMan
Guerilla Ontologist

Joined: 01 Jul 2003
Posts: 13143

Thanks, I knew I probably wasn't using the right tools. On the surface, it would seem sed would be capable of what I was needing. Unfortunately, sed does not support variables. I've never used awk, and just looking at what you posted, I'm able to make sense of the syntax. I'll definitely check out awk some more. I might even look for a copy of that book.
_________________
Blargle Flarg Bergah Merg

Wed Jan 13, 2010 4:13 pm

LightningCrash
Smile like Bob, order your free LC today

Joined: 03 Apr 2003
Posts: 5020

TehDanMan wrote: Thanks, I knew I probably wasn't using the right tools. On the surface, it would seem sed would be capable of what I was needing. Unfortunately, sed does not support variables. I've never used awk, and just looking at what you posted, I'm able to make sense of the syntax. I'll definitely check out awk some more. I might even look for a copy of that book.

Awk was designed to process records. Everything is a record. It is very much at home when there are fields and field separators. It makes it very useful for anything to do with /etc/passwd or htpasswd files.
I've seen that book all over in many book collections. It is also a common target for piracy and is ubiquitous on popular piracy sites. Used copies are cheap.

But don't count out sed! Sed sorta supports variables. Sed will let you dick with the matched pattern afterwards. You can do it with sed, it's just harder.

Look at this:

Code




lc@laptop:~/tmp$ cat test|sed 's/^\(\w*\):.*/\1:\1/'


user1:user1


user2:user2


user3:user3

explaining the matching:

Code




^            (beginning of the line)


\(           (start a pattern match)


\w*         (match non-whitespace, the asterisk is for "one or more")


\)           (end a pattern match)


:            (literal colon)


.*           (everything else; eg one or more of any character at all)

on the substitution side:

Code




\1         (reprint the first pattern)


:           (literal colon)


\1          (reprint the first pattern)

There are a million ways to skin that cat, none of them are necessarily wrong. Very Happy

Wed Jan 13, 2010 7:54 pm

TehDanMan
Guerilla Ontologist

Joined: 01 Jul 2003
Posts: 13143

Well, despite my best efforts to fuck it up. I've set up a pretty robust squid system. Maybe I'll post a howto or something.
_________________
Blargle Flarg Bergah Merg

Fri Jan 15, 2010 3:32 pm

TehDanMan
Guerilla Ontologist

Joined: 01 Jul 2003
Posts: 13143

Hah! I'm so awesome, I completed another big project today.
_________________
Blargle Flarg Bergah Merg

Fri Jan 15, 2010 4:12 pm

LightningCrash
Smile like Bob, order your free LC today

Joined: 03 Apr 2003
Posts: 5020

you used the username as the password in an ncsa_auth DB, didn't you?

Fri Jan 15, 2010 4:37 pm

TehDanMan
Guerilla Ontologist

Joined: 01 Jul 2003
Posts: 13143

No, the need to translate the username:username file was needed in order to generate human readable reporting. If I'd used sarg, I wouldn't have had the problem. However, a third-party vendor gives them all sorts of reporting info and it was possible to tie-in squid reports if I had that file. It saved them a few grand not to have this vendor provide their proxy server.
_________________
Blargle Flarg Bergah Merg

Fri Jan 15, 2010 5:06 pm

Display posts from previous:

< Last Thread | Next Thread >

Quick Reply

Forum Rules:

You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum