Ygtjki

How to format long polynomial?

"You are your self first supporter", a more proper way to say it

I'm flying to France today and my passport expires in less than 2 months

NMaximize is not converging to a solution

Rock identification in KY

Which country benefited the most from UN Security Council vetoes?

What's the output of a record needle playing an out-of-speed record

Is it legal for company to use my work email to pretend I still work there?

What defenses are there against being summoned by the Gate spell?

How is the claim "I am in New York only if I am in America" the same as "If I am in New York, then I am in America?

Can an x86 CPU running in real mode be considered to be basically an 8086 CPU?

How can I make my BBEG immortal short of making them a Lich or Vampire?

Horror movie about a virus at the prom; beginning and end are stylized as a cartoon

Today is the Center

Was any UN Security Council vote triple-vetoed?

Client team has low performances and low technical skills: we always fix their work and now they stop collaborate with us. How to solve?

How does quantile regression compare to logistic regression with the variable split at the quantile?

How to determine what difficulty is right for the game?

Add text to same line using sed

Is it unprofessional to ask if a job posting on GlassDoor is real?

Theorems that impeded progress

What does the "remote control" for a QF-4 look like?

Are astronomers waiting to see something in an image from a gravitational lens that they've already seen in an adjacent image?

Is it possible to do 50 km distance without any previous training?

Remove duplicate lines from a file but leave 1 occurrence

Remove duplicate lines while keeping the order of the linesHow can I “cat” a file and remove commented lines?How can I write a sed script to delete numbers from a line?Merge labels from lines with duplicate fieldsHow can i remove duplicates from the output of tshark outputDelete all lines that contain duplicate lettersprint the duplicate lines only on fields 1,2 from csv fileremove lines in file 1 from file 2how to verify occurrence of 3 lines in filePrint non consecutive lines from file given line number listRemove specific words within lines

.everyoneloves__top-leaderboard:empty,.everyoneloves__mid-leaderboard:empty,.everyoneloves__bot-mid-leaderboard:empty margin-bottom:0;

0

I'm looking to remove duplicate lines from a file but leave 1 occurrence in the file.

Example of the file:

this is a string
test line
test line 2
this is a string

From the above example, I would want to remove 1 occurrence of "this is a string".

Best way to do this?

linux

share|improve this question

edited May 19 '18 at 18:33

Tom Bailey

asked May 19 '18 at 13:09

Tom BaileyTom Bailey

162

bumped to the homepage by Community♦ 2 days ago

This question has answers that may be good or bad; the system has marked it active so that they can be reviewed.

• 
  
  
  

  
  1
  

  
  
  
  
  
  

  
  
  With such questions you should always provide example input and output.
  
  – Hauke Laging
  May 19 '18 at 13:12
  

  
  
  
  


• 
  
  
  

  
  1
  

  
  
  
  
  
  

  
  
  Possibly related: Remove duplicate lines while keeping the order of the lines
  
  – steeldriver
  May 19 '18 at 13:12
  

  
  
  
  


• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  Are the duplicated lines adjacent to one another? Is the output to remain in the same order or would it be ok to sort the data?
  
  – Kusalananda♦
  May 19 '18 at 13:14
  

  
  
  
  


• 
  
  
  

  
  1
  

  
  
  
  
  
  

  
  
  Keep one occurrence of a duplicate (ie two identical lines per match) or simply "remove all duplicate lines, leaving only one line per set of duplicates"? Does the final order matter?
  
  – roaima
  May 19 '18 at 13:17
  

  
  
  
  


• 
  
  
  

  
  1
  

  
  
  
  
  
  

  
  
  it is not a problem for you that the lines will be sorted, then a sort file|uniq will do what you want.
  
  – peterh
  May 19 '18 at 19:03
  

  
  
  
  

 | 
show 4 more comments

0

I'm looking to remove duplicate lines from a file but leave 1 occurrence in the file.

Example of the file:

this is a string
test line
test line 2
this is a string

From the above example, I would want to remove 1 occurrence of "this is a string".

Best way to do this?

linux

share|improve this question

edited May 19 '18 at 18:33

Tom Bailey

asked May 19 '18 at 13:09

Tom BaileyTom Bailey

162

bumped to the homepage by Community♦ 2 days ago

This question has answers that may be good or bad; the system has marked it active so that they can be reviewed.

• 
  
  
  

  
  1
  

  
  
  
  
  
  

  
  
  With such questions you should always provide example input and output.
  
  – Hauke Laging
  May 19 '18 at 13:12
  

  
  
  
  


• 
  
  
  

  
  1
  

  
  
  
  
  
  

  
  
  Possibly related: Remove duplicate lines while keeping the order of the lines
  
  – steeldriver
  May 19 '18 at 13:12
  

  
  
  
  


• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  Are the duplicated lines adjacent to one another? Is the output to remain in the same order or would it be ok to sort the data?
  
  – Kusalananda♦
  May 19 '18 at 13:14
  

  
  
  
  


• 
  
  
  

  
  1
  

  
  
  
  
  
  

  
  
  Keep one occurrence of a duplicate (ie two identical lines per match) or simply "remove all duplicate lines, leaving only one line per set of duplicates"? Does the final order matter?
  
  – roaima
  May 19 '18 at 13:17
  

  
  
  
  


• 
  
  
  

  
  1
  

  
  
  
  
  
  

  
  
  it is not a problem for you that the lines will be sorted, then a sort file|uniq will do what you want.
  
  – peterh
  May 19 '18 at 19:03
  

  
  
  
  

 | 
show 4 more comments

I'm looking to remove duplicate lines from a file but leave 1 occurrence in the file.

Example of the file:

this is a string
test line
test line 2
this is a string

From the above example, I would want to remove 1 occurrence of "this is a string".

Best way to do this?

linux

share|improve this question

edited May 19 '18 at 18:33

Tom Bailey

asked May 19 '18 at 13:09

Tom BaileyTom Bailey

162

this is a string
test line
test line 2
this is a string

share|improve this question

edited May 19 '18 at 18:33

Tom Bailey

asked May 19 '18 at 13:09

Tom BaileyTom Bailey

162

share|improve this question

edited May 19 '18 at 18:33

Tom Bailey

asked May 19 '18 at 13:09

Tom BaileyTom Bailey

162

asked May 19 '18 at 13:09

Tom BaileyTom Bailey

162

asked May 19 '18 at 13:09

Tom BaileyTom Bailey

162

2 Answers
2

active

oldest

votes

1

Demo file stuff.txt contains:

one
two
three
one
two
four
five

Remove duplicate lines from a file assuming you don't mind that lines are sorted

$ sort -u stuff.txt 
five
four
one
three
two

Explanation: the u flag sent to sort says sort the lines of the file and force unique.

Remove duplicate lines from a file, preserve original ordering, keep the first:

$ cat -n stuff.txt | sort -uk2 | sort -nk1 | cut -f2-
one
two
three
four
five

Explanation: The n flag passed to cat appends line numbers to left of every line, plus space, then the first sort says sort by unique and but only after the first word, the second sort command says use the line numbers we stored in step 1 to resort by the original ordering, finally cut off the first word.

Remove duplicate lines from a file, preserve order, keep last.

tac stuff.txt > stuff2.txt; cat -n stuff2.txt | sort -uk2 | sort -nk1 | cut -f2- > stuff3.txt; tac stuff3.txt > stuff4.txt; cat stuff4.txt
three
one
two
four
five

Explanation: Same as before, but tac reverse the file, achieving the desired result.

share|improve this answer

answered Mar 3 at 5:33

Eric LeschinskiEric Leschinski

1,35711416

add a comment | 

0

This leaves the first occurrence:

awk '! a[$0]++' inputfile

start cmd:> echo 'this is a string
cont. cmd:> test line
cont. cmd:> test line 2
cont. cmd:> this is a string' | awk '! a[$0]++'
 this is a string
 test line
 test line 2

share|improve this answer

edited May 19 '18 at 20:16

answered May 19 '18 at 13:16

Hauke LagingHauke Laging

57.7k1287137

• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  It seems to just print out and not actually make in changes in the file.
  
  – Tom Bailey
  May 19 '18 at 15:49
  

  
  
  
  


• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  @TomBailey That's why I told you to provide example input and output. I did test it and it works fine for me.
  
  – Hauke Laging
  May 19 '18 at 16:49
  
  

  
  
  
  


• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  I have edited it now.
  
  – Tom Bailey
  May 19 '18 at 19:29
  

  
  
  
  


• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  @TomBailey works fine for me.
  
  – Hauke Laging
  May 19 '18 at 20:16
  

  
  
  
  

add a comment | 

Your Answer

StackExchange.ready(function()
var channelOptions =
tags: "".split(" "),
id: "106"
;
initTagRenderer("".split(" "), "".split(" "), channelOptions);

StackExchange.using("externalEditor", function()
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled)
StackExchange.using("snippets", function()
createEditor();
);

else
createEditor();

);

function createEditor()
StackExchange.prepareEditor(
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: false,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: null,
bindNavPrevention: true,
postfix: "",
imageUploader:
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
,
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
);



);

draft saved

draft discarded

Sign up or log in

StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);

Sign up using Google

Sign up using Facebook

Sign up using Email and Password

Post as a guest

Name

Email

Required, but never shown

StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2funix.stackexchange.com%2fquestions%2f444795%2fremove-duplicate-lines-from-a-file-but-leave-1-occurrence%23new-answer', 'question_page');

);

Post as a guest

Name

Email

Required, but never shown

1

Demo file stuff.txt contains:

one
two
three
one
two
four
five

Remove duplicate lines from a file assuming you don't mind that lines are sorted

$ sort -u stuff.txt 
five
four
one
three
two

Explanation: the u flag sent to sort says sort the lines of the file and force unique.

Remove duplicate lines from a file, preserve original ordering, keep the first:

$ cat -n stuff.txt | sort -uk2 | sort -nk1 | cut -f2-
one
two
three
four
five

Explanation: The n flag passed to cat appends line numbers to left of every line, plus space, then the first sort says sort by unique and but only after the first word, the second sort command says use the line numbers we stored in step 1 to resort by the original ordering, finally cut off the first word.

Remove duplicate lines from a file, preserve order, keep last.

tac stuff.txt > stuff2.txt; cat -n stuff2.txt | sort -uk2 | sort -nk1 | cut -f2- > stuff3.txt; tac stuff3.txt > stuff4.txt; cat stuff4.txt
three
one
two
four
five

Explanation: Same as before, but tac reverse the file, achieving the desired result.

share|improve this answer

answered Mar 3 at 5:33

Eric LeschinskiEric Leschinski

1,35711416

add a comment | 

1

Demo file stuff.txt contains:

one
two
three
one
two
four
five

Remove duplicate lines from a file assuming you don't mind that lines are sorted

$ sort -u stuff.txt 
five
four
one
three
two

Explanation: the u flag sent to sort says sort the lines of the file and force unique.

Remove duplicate lines from a file, preserve original ordering, keep the first:

$ cat -n stuff.txt | sort -uk2 | sort -nk1 | cut -f2-
one
two
three
four
five

Explanation: The n flag passed to cat appends line numbers to left of every line, plus space, then the first sort says sort by unique and but only after the first word, the second sort command says use the line numbers we stored in step 1 to resort by the original ordering, finally cut off the first word.

Remove duplicate lines from a file, preserve order, keep last.

tac stuff.txt > stuff2.txt; cat -n stuff2.txt | sort -uk2 | sort -nk1 | cut -f2- > stuff3.txt; tac stuff3.txt > stuff4.txt; cat stuff4.txt
three
one
two
four
five

Explanation: Same as before, but tac reverse the file, achieving the desired result.

share|improve this answer

answered Mar 3 at 5:33

Eric LeschinskiEric Leschinski

1,35711416

add a comment | 

Demo file stuff.txt contains:

one
two
three
one
two
four
five

Remove duplicate lines from a file assuming you don't mind that lines are sorted

$ sort -u stuff.txt 
five
four
one
three
two

Explanation: the u flag sent to sort says sort the lines of the file and force unique.

Remove duplicate lines from a file, preserve original ordering, keep the first:

$ cat -n stuff.txt | sort -uk2 | sort -nk1 | cut -f2-
one
two
three
four
five

Explanation: The n flag passed to cat appends line numbers to left of every line, plus space, then the first sort says sort by unique and but only after the first word, the second sort command says use the line numbers we stored in step 1 to resort by the original ordering, finally cut off the first word.

Remove duplicate lines from a file, preserve order, keep last.

tac stuff.txt > stuff2.txt; cat -n stuff2.txt | sort -uk2 | sort -nk1 | cut -f2- > stuff3.txt; tac stuff3.txt > stuff4.txt; cat stuff4.txt
three
one
two
four
five

Explanation: Same as before, but tac reverse the file, achieving the desired result.

share|improve this answer

answered Mar 3 at 5:33

Eric LeschinskiEric Leschinski

1,35711416

one
two
three
one
two
four
five

$ sort -u stuff.txt 
five
four
one
three
two

$ cat -n stuff.txt | sort -uk2 | sort -nk1 | cut -f2-
one
two
three
four
five

tac stuff.txt > stuff2.txt; cat -n stuff2.txt | sort -uk2 | sort -nk1 | cut -f2- > stuff3.txt; tac stuff3.txt > stuff4.txt; cat stuff4.txt
three
one
two
four
five

share|improve this answer

answered Mar 3 at 5:33

Eric LeschinskiEric Leschinski

1,35711416

answered Mar 3 at 5:33

Eric LeschinskiEric Leschinski

1,35711416

answered Mar 3 at 5:33

Eric LeschinskiEric Leschinski

1,35711416

0

This leaves the first occurrence:

awk '! a[$0]++' inputfile

start cmd:> echo 'this is a string
cont. cmd:> test line
cont. cmd:> test line 2
cont. cmd:> this is a string' | awk '! a[$0]++'
 this is a string
 test line
 test line 2

share|improve this answer

edited May 19 '18 at 20:16

answered May 19 '18 at 13:16

Hauke LagingHauke Laging

57.7k1287137

• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  It seems to just print out and not actually make in changes in the file.
  
  – Tom Bailey
  May 19 '18 at 15:49
  

  
  
  
  


• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  @TomBailey That's why I told you to provide example input and output. I did test it and it works fine for me.
  
  – Hauke Laging
  May 19 '18 at 16:49
  
  

  
  
  
  


• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  I have edited it now.
  
  – Tom Bailey
  May 19 '18 at 19:29
  

  
  
  
  


• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  @TomBailey works fine for me.
  
  – Hauke Laging
  May 19 '18 at 20:16
  

  
  
  
  

add a comment | 

0

This leaves the first occurrence:

awk '! a[$0]++' inputfile

start cmd:> echo 'this is a string
cont. cmd:> test line
cont. cmd:> test line 2
cont. cmd:> this is a string' | awk '! a[$0]++'
 this is a string
 test line
 test line 2

share|improve this answer

edited May 19 '18 at 20:16

answered May 19 '18 at 13:16

Hauke LagingHauke Laging

57.7k1287137

• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  It seems to just print out and not actually make in changes in the file.
  
  – Tom Bailey
  May 19 '18 at 15:49
  

  
  
  
  


• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  @TomBailey That's why I told you to provide example input and output. I did test it and it works fine for me.
  
  – Hauke Laging
  May 19 '18 at 16:49
  
  

  
  
  
  


• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  I have edited it now.
  
  – Tom Bailey
  May 19 '18 at 19:29
  

  
  
  
  


• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  @TomBailey works fine for me.
  
  – Hauke Laging
  May 19 '18 at 20:16
  

  
  
  
  

add a comment | 

This leaves the first occurrence:

awk '! a[$0]++' inputfile

start cmd:> echo 'this is a string
cont. cmd:> test line
cont. cmd:> test line 2
cont. cmd:> this is a string' | awk '! a[$0]++'
 this is a string
 test line
 test line 2

share|improve this answer

edited May 19 '18 at 20:16

answered May 19 '18 at 13:16

Hauke LagingHauke Laging

57.7k1287137

awk '! a[$0]++' inputfile

start cmd:> echo 'this is a string
cont. cmd:> test line
cont. cmd:> test line 2
cont. cmd:> this is a string' | awk '! a[$0]++'
 this is a string
 test line
 test line 2

share|improve this answer

edited May 19 '18 at 20:16

answered May 19 '18 at 13:16

Hauke LagingHauke Laging

57.7k1287137

answered May 19 '18 at 13:16

Hauke LagingHauke Laging

57.7k1287137

answered May 19 '18 at 13:16

Hauke LagingHauke Laging

57.7k1287137