asked    Lesley     2018-10-22       bash       118 view        2 Answers

[SOLVED] How to use AWK with string as a RS?

I want to use AWK, but I don't seem to get the first record right. I hope anyone can help to get it right.

I have this file, every record is 3 lines but sometimes it has 4 lines (so there is a $3 and $4). My goal is to print all three lines of each record, and if there is a forth line I want also to print the first 2 lines with the forth (without the 3rd).

My strategy is to use a string ("Sequence: ") as a RS, and new line ("\n") for FS.

My file looks like this:

Sequence: X92272_IGHV4-31*08_Homosapiens_F_V-REGION_140..429_290nt_1_____290+0=290_partialin3'__     from: 1   to: 290
Start     End  Strand Pattern                 Mismatch Sequence
184     192       + pattern:AA[CT]NNN[AT]CN        . aacccgtcc
Sequence: X92273_IGHV4-31*09_Homosapiens_F_V-REGION_140..429_290nt_1_____290+0=290_partialin3'__     from: 1   to: 290
Start     End  Strand Pattern                 Mismatch Sequence
184     192       + pattern:AA[CT]NNN[AT]CN        . aacccgtcc
Sequence: Z14235_IGHV4-31*10_Homosapiens_F_V-REGION_140..438_299nt_1_____299+0=299___     from: 1   to: 299
Start     End  Strand Pattern                 Mismatch Sequence
184     192       + pattern:AA[CT]NNN[AT]CN        . aacccgtcc
Sequence: AB019439_IGHV4-34*01_Homosapiens_F_V-REGION_59657..59949_293nt_1_____293+0=293___     from: 1   to: 293
Start     End  Strand Pattern                 Mismatch Sequence
150     158       + pattern:AA[CT]NNN[AT]CN        . aatcaatca
178     186       + pattern:AA[CT]NNN[AT]CN        . aacccgtcc
Sequence: M99684_IGHV4-34*02_Homosapiens_F_V-REGION_311..603_293nt_1_____293+0=293___     from: 1   to: 293
Start     End  Strand Pattern                 Mismatch Sequence
150     158       + pattern:AA[CT]NNN[AT]CN        . aatcaatca
178     186       + pattern:AA[CT]NNN[AT]CN        . aacccgtcc

With the following code I get a messed up first record, because the string is in the beginning of the file as well.

awk '{ RS="Sequence: "; FS="\n" }
{
if ($4 != "" )
    print $1,"\n",$2,"\n",$3,"\n",$1,"\n",$2,"\n",$4
else
    print $1,"\n",$2,"\n",$3 ;
}' short.txt > test 

With output:

Sequence:
 X92272_IGHV4-31*08_Homosapiens_F_V-REGION_140..429_290nt_1_____290+0=290_partialin3'__
 from:
 Sequence:
 X92272_IGHV4-31*08_Homosapiens_F_V-REGION_140..429_290nt_1_____290+0=290_partialin3'__
 1
Start     End  Strand Pattern                 Mismatch Sequence
 184     192       + pattern:AA[CT]NNN[AT]CN        . aacccgtcc

X92273_IGHV4-31*09_Homosapiens_F_V-REGION_140..429_290nt_1_____290+0=290_partialin3'__     from: 1   to: 290
 Start     End  Strand Pattern                 Mismatch Sequence
 184     192       + pattern:AA[CT]NNN[AT]CN        . aacccgtcc
Z14235_IGHV4-31*10_Homosapiens_F_V-REGION_140..438_299nt_1_____299+0=299___     from: 1   to: 299
 Start     End  Strand Pattern                 Mismatch Sequence
 184     192       + pattern:AA[CT]NNN[AT]CN        . aacccgtcc
AB019439_IGHV4-34*01_Homosapiens_F_V-REGION_59657..59949_293nt_1_____293+0=293___     from: 1   to: 293
 Start     End  Strand Pattern                 Mismatch Sequence
 150     158       + pattern:AA[CT]NNN[AT]CN        . aatcaatca
 AB019439_IGHV4-34*01_Homosapiens_F_V-REGION_59657..59949_293nt_1_____293+0=293___     from: 1   to: 293
 Start     End  Strand Pattern                 Mismatch Sequence
 178     186       + pattern:AA[CT]NNN[AT]CN        . aacccgtcc
M99684_IGHV4-34*02_Homosapiens_F_V-REGION_311..603_293nt_1_____293+0=293___     from: 1   to: 293
 Start     End  Strand Pattern                 Mismatch Sequence
 150     158       + pattern:AA[CT]NNN[AT]CN        . aatcaatca
 M99684_IGHV4-34*02_Homosapiens_F_V-REGION_311..603_293nt_1_____293+0=293___     from: 1   to: 293
 Start     End  Strand Pattern                 Mismatch Sequence
 178     186       + pattern:AA[CT]NNN[AT]CN        . aacccgtcc

So I thought I should remove the first "Sequence: " string from the input file, but that gives:

X92272_IGHV4-31*08_Homosapiens_F_V-REGION_140..429_290nt_1_____290+0=290_partialin3'__
 from:
 1
 X92272_IGHV4-31*08_Homosapiens_F_V-REGION_140..429_290nt_1_____290+0=290_partialin3'__
 from:
 to:
Start     End  Strand Pattern                 Mismatch Sequence
 184     192       + pattern:AA[CT]NNN[AT]CN        . aacccgtcc

X92273_IGHV4-31*09_Homosapiens_F_V-REGION_140..429_290nt_1_____290+0=290_partialin3'__     from: 1   to: 290
 Start     End  Strand Pattern                 Mismatch Sequence
 184     192       + pattern:AA[CT]NNN[AT]CN        . aacccgtcc
Z14235_IGHV4-31*10_Homosapiens_F_V-REGION_140..438_299nt_1_____299+0=299___     from: 1   to: 299
 Start     End  Strand Pattern                 Mismatch Sequence
 184     192       + pattern:AA[CT]NNN[AT]CN        . aacccgtcc
AB019439_IGHV4-34*01_Homosapiens_F_V-REGION_59657..59949_293nt_1_____293+0=293___     from: 1   to: 293
 Start     End  Strand Pattern                 Mismatch Sequence
 150     158       + pattern:AA[CT]NNN[AT]CN        . aatcaatca
 AB019439_IGHV4-34*01_Homosapiens_F_V-REGION_59657..59949_293nt_1_____293+0=293___     from: 1   to: 293
 Start     End  Strand Pattern                 Mismatch Sequence
 178     186       + pattern:AA[CT]NNN[AT]CN        . aacccgtcc
M99684_IGHV4-34*02_Homosapiens_F_V-REGION_311..603_293nt_1_____293+0=293___     from: 1   to: 293
 Start     End  Strand Pattern                 Mismatch Sequence
 150     158       + pattern:AA[CT]NNN[AT]CN        . aatcaatca
 M99684_IGHV4-34*02_Homosapiens_F_V-REGION_311..603_293nt_1_____293+0=293___     from: 1   to: 293
 Start     End  Strand Pattern                 Mismatch Sequence
 178     186       + pattern:AA[CT]NNN[AT]CN        . aacccgtcc

So again the first record is messed up. Is there a solution to this problem? My expected output is as the last output (with or without the string "Sequence :"), but with the first record correct.

  2 Answers  

        answered    Lee     2018-10-22      

It sounds like this is what you're trying to do:

$ cat tst.awk
/^Sequence/ { if (NR>1) prt() }
{ rec[++cnt] = $0 }
END { prt() }
function prt() {
    print rec[1] ORS rec[2] ORS rec[3]
    if (cnt == 4) {
        print rec[1] ORS rec[2] ORS rec[4]
    }
    cnt=0
}

$ awk -f tst.awk file
Sequence: X92272_IGHV4-31*08_Homosapiens_F_V-REGION_140..429_290nt_1_____290+0=290_partialin3'__     from: 1   to: 290
Start     End  Strand Pattern                 Mismatch Sequence
184     192       + pattern:AA[CT]NNN[AT]CN        . aacccgtcc
Sequence: X92273_IGHV4-31*09_Homosapiens_F_V-REGION_140..429_290nt_1_____290+0=290_partialin3'__     from: 1   to: 290
Start     End  Strand Pattern                 Mismatch Sequence
184     192       + pattern:AA[CT]NNN[AT]CN        . aacccgtcc
Sequence: Z14235_IGHV4-31*10_Homosapiens_F_V-REGION_140..438_299nt_1_____299+0=299___     from: 1   to: 299
Start     End  Strand Pattern                 Mismatch Sequence
184     192       + pattern:AA[CT]NNN[AT]CN        . aacccgtcc
Sequence: AB019439_IGHV4-34*01_Homosapiens_F_V-REGION_59657..59949_293nt_1_____293+0=293___     from: 1   to: 293
Start     End  Strand Pattern                 Mismatch Sequence
150     158       + pattern:AA[CT]NNN[AT]CN        . aatcaatca
Sequence: AB019439_IGHV4-34*01_Homosapiens_F_V-REGION_59657..59949_293nt_1_____293+0=293___     from: 1   to: 293
Start     End  Strand Pattern                 Mismatch Sequence
178     186       + pattern:AA[CT]NNN[AT]CN        . aacccgtcc
Sequence: M99684_IGHV4-34*02_Homosapiens_F_V-REGION_311..603_293nt_1_____293+0=293___     from: 1   to: 293
Start     End  Strand Pattern                 Mismatch Sequence
150     158       + pattern:AA[CT]NNN[AT]CN        . aatcaatca
Sequence: M99684_IGHV4-34*02_Homosapiens_F_V-REGION_311..603_293nt_1_____293+0=293___     from: 1   to: 293
Start     End  Strand Pattern                 Mismatch Sequence
178     186       + pattern:AA[CT]NNN[AT]CN        . aacccgtcc

Trying to use a RS for this just makes your life harder and the resulting code non-portable (gawk-only)



        answered    Cash     2018-10-22      

Your code can be easily fixed as:

BEGIN{ RS="Sequence: "; FS="\n" }
(NR==1){next}
{
if ($4 != "" )
    print $1,"\n",$2,"\n",$3,"\n",$1,"\n",$2,"\n",$4
else
    print $1,"\n",$2,"\n",$3 ;
}

The first record would be empty, that is why it is skipped with next.

The reason you had problems with your first record is that you defined RS and FS after the first record was read (i.e. not in a BEGIN block which occurs before anything is done at all)

But what you really want, just to be sure, is RS="(^|\n)Sequence: " This just to be sure that it starts at the beginning of the line or the file.





Your Answer





 2018-10-22         Enoch

SQL Relationship to Secondary Table Without Adding Duplicate Column

There are Cities, Job Types, and Tasks. Cities can have multiple Job Types. Tasks are created for a City that can have multiple Job Types. Cities can have Many Tasks. But when adding the Job Types for the Tasks assigned to a City, must make sure the City has that Job Type. How to create Relationship/Constraint when Adding/Updating Task_JobTypes to make sure the City associated with the Task has that Job Type allowed in City_JobTypes? The constraint in Task_JobTypes "FK_Task_JobTypes_JobTypes" need to be referencing that instead of just JobTypes.Cities - Id,Name JobType...
 sql                     2 answers                     60 view
 2018-10-22         Noah

SQL - using calculated dates in WHERE clause

I have a table with future values where I only want the last full week. Meaning af want to find the last Sunday and have a WHERE clause where date is between 'last Sunday - 7' and 'last Sunday'. Meaning I have transactions until '31-10-2018' and want the period between '2018-10-28' and '2018-10-22' (both days included). But every week new data will be added continuously so very soon I have data until '2018-11-30' and so on.I have tried a query like:SELECT *FROM PlannedCountsWHERE [Date] BETWEEN DATEADD(d,-6,MAX([Date])) and MAX([Date])Note: I'm aware that this will only giv...
 sql                     2 answers                     61 view
 2018-10-22         Marcus

Sqlite getting the latest record for this table

I can't figure out what I am doing wrong here. I am querying sqlite table the to get the latest record for each College but the result is not completely accurate. Below is the source table:College Status Date"UTAR "CERTIFIED" "07/30/2018""UTA" "CERTIFIED" "03/19/2018""UTD" "CERTIFIED" "06/08/2018""UTEL" "CERTIFIED" "03/13/2018""UTSA" "CERTIFIED" "02/06/2018""UTT" "CERTIFIED" "03/14/2018""UTPB" "CERTIFIED" "07/23/2018""UTRG" "CERTIFIED" "04/12/2018""UTAR" "CERTIFIED" "05/31/2017""UTA" "CERTIFIED" "03/02/2017""UTD" "CERTIFIED" "04/13/2017""UTEL" "...
 sql                     2 answers                     108 view