Sfrgttk

How to avoid being sexist when trying to employ someone to function in a very sexist environment?

Where is the fallacy here?

How would we write a misogynistic character without offending people?

What am I? I am in theaters and computer programs

What is better: yes / no radio, or simple checkbox?

How to add multiple differently colored borders around a node?

How to approximate rolls for potions of healing using only d6's?

Equivalent to "source" in OpenBSD?

Whom do I have to contact for a ticket refund in case of denied boarding (in the EU)?

Difference between 'stomach' and 'uterus'

How do ISS astronauts "get their stripes"?

Should I choose Itemized or Standard deduction?

What's the purpose of these copper coils with resistors inside them in A Yamaha RX-V396RDS amplifier?

Why do members of Congress in committee hearings ask witnesses the same question multiple times?

How to deny access to SQL Server to certain login over SSMS, but allow over .Net SqlClient Data Provider

Which aircraft had such a luxurious-looking navigator's station?

chrony vs. systemd-timesyncd – What are the differences and use cases as NTP clients?

How to count words in a line

How to speed up a process

What is a term for a function that when called repeatedly, has the same effect as calling once?

Six real numbers so that product of any five is the sixth one

What to do when being responsible for data protection in your lab, yet advice is ignored?

Pure Functions: Does "No Side Effects" Imply "Always Same Output, Given Same Input"?

Where is this triangular-shaped space station from?

MongoDB merge two collections and drop documents with same value in field

MongoDB - handling embedded documents and relationsMongodb extremely slow with bulk inserts and showing erratic behaviorSeparate login for user / administrators using mongodbMongo Chunk 250000 document limit is creating problem in chunk migrationMongoDB document size for collection - impact on RAM and query performanceHow unique are MongoDB _id fields?Adding a Mongo Shard decreases the insert performanceIs it possible which documents couldn't be dumped using “mongodump”?Reliability of mongodumpMongoDB Zone Sharding without min & max key

1

In my database I have two collections, but some documents were added (possibly) at different times to both collections.

I could use mongodump and then mongorestore for merging. But then I have the same documents that were added to both collections as duplicates in my new collection. mongorestore --drop does not help neither, because the documents not necessarily have the same _id.

How to drop a document when a document with userid exists already?

mongodb mongorestore

share|improve this question

edited Nov 27 '18 at 14:50

Wuff

asked Nov 27 '18 at 14:28

WuffWuff

62

bumped to the homepage by Community♦ 4 mins ago

This question has answers that may be good or bad; the system has marked it active so that they can be reviewed.

• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  Welcome to the StackExchange. what is MongoDB version(x,y,z)?
  
  – Md Haidar Ali Khan
  Nov 29 '18 at 9:23
  

  
  
  
  


• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  I use MongoDB v3.6.5 (build environment: distmod: debian92 distarch: x86_64 target_arch: x86_64)
  
  – Wuff
  Dec 12 '18 at 12:43
  

  
  
  
  

add a comment | 

1

In my database I have two collections, but some documents were added (possibly) at different times to both collections.

I could use mongodump and then mongorestore for merging. But then I have the same documents that were added to both collections as duplicates in my new collection. mongorestore --drop does not help neither, because the documents not necessarily have the same _id.

How to drop a document when a document with userid exists already?

mongodb mongorestore

share|improve this question

edited Nov 27 '18 at 14:50

Wuff

asked Nov 27 '18 at 14:28

WuffWuff

62

bumped to the homepage by Community♦ 4 mins ago

This question has answers that may be good or bad; the system has marked it active so that they can be reviewed.

• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  Welcome to the StackExchange. what is MongoDB version(x,y,z)?
  
  – Md Haidar Ali Khan
  Nov 29 '18 at 9:23
  

  
  
  
  


• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  I use MongoDB v3.6.5 (build environment: distmod: debian92 distarch: x86_64 target_arch: x86_64)
  
  – Wuff
  Dec 12 '18 at 12:43
  

  
  
  
  

add a comment | 

In my database I have two collections, but some documents were added (possibly) at different times to both collections.

I could use mongodump and then mongorestore for merging. But then I have the same documents that were added to both collections as duplicates in my new collection. mongorestore --drop does not help neither, because the documents not necessarily have the same _id.

How to drop a document when a document with userid exists already?

mongodb mongorestore

share|improve this question

edited Nov 27 '18 at 14:50

Wuff

asked Nov 27 '18 at 14:28

WuffWuff

62

share|improve this question

edited Nov 27 '18 at 14:50

Wuff

asked Nov 27 '18 at 14:28

WuffWuff

62

share|improve this question

edited Nov 27 '18 at 14:50

Wuff

asked Nov 27 '18 at 14:28

WuffWuff

62

asked Nov 27 '18 at 14:28

WuffWuff

62

asked Nov 27 '18 at 14:28

WuffWuff

62

1 Answer
1

active

oldest

votes

0

There might be more elegent solutions for this, but let me answer how I merged a new collection into an original collection without importing the duplicates.

1. Compare userid fields in both collections and take the difference.


2. Use mongos $out operator to save documents to a difference collection. This new collection has all documents of the new collection, except of the duplicates.


3. 
   mongodump the difference collection and mongorestore it into the original collection.

from pymongo import MongoClient

import subprocess



# collection names

db_name = 'db'

col_original = 'col_original'

col_new = 'col_new'

field = 'userid'





# Get ids from collection

def get_ids(db, col):

    docs = db[col].aggregate([

        {'$project':

         {'_id': 0,

          'id': '$' + field}}

    ])

    docs = list(docs)

    ids = [x['id'] for x in docs]

    return ids





# Connect to MongoDB

client = MongoClient('mongodb://localhost')

db = client[db_name]



# Get difference in ids

ids_new = get_ids(db, col_new)

ids_original = get_ids(db, col_original)

ids_diff = list(set(ids_new).difference(ids_original))



# Get all documents with userid that are in col_new, but not in

# col_original. Hence, all duplicates are skipped.

db[col_new].aggregate([

    {'$match': {

        field: {'$in': ids_diff}}},

    {'$out': 'col_diff'}])



# Use mongodump to save col_diff

subprocess.check_output(['mongodump',

                         '-d',

                         db_name,

                         '-c',

                         'col_diff'])



# Merge col_diff into col_original

subprocess.check_output(['mongorestore',

                         '-d',

                         db_name,

                         '-c',

                         col_original,

                         'dump/' + db_name + '/col_diff.bson'])

share|improve this answer

answered Dec 12 '18 at 12:41

WuffWuff

62

add a comment | 

Your Answer

StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "182"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);

StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});

function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: false,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: null,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});


}
});

draft saved

draft discarded

Sign up or log in

StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});

Sign up using Google

Sign up using Facebook

Sign up using Email and Password

Post as a guest

Name

Email

Required, but never shown

StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fdba.stackexchange.com%2fquestions%2f223551%2fmongodb-merge-two-collections-and-drop-documents-with-same-value-in-field%23new-answer', 'question_page');
}
);

Post as a guest

Name

Email

Required, but never shown

0

There might be more elegent solutions for this, but let me answer how I merged a new collection into an original collection without importing the duplicates.

1. Compare userid fields in both collections and take the difference.


2. Use mongos $out operator to save documents to a difference collection. This new collection has all documents of the new collection, except of the duplicates.


3. 
   mongodump the difference collection and mongorestore it into the original collection.

from pymongo import MongoClient

import subprocess



# collection names

db_name = 'db'

col_original = 'col_original'

col_new = 'col_new'

field = 'userid'





# Get ids from collection

def get_ids(db, col):

    docs = db[col].aggregate([

        {'$project':

         {'_id': 0,

          'id': '$' + field}}

    ])

    docs = list(docs)

    ids = [x['id'] for x in docs]

    return ids





# Connect to MongoDB

client = MongoClient('mongodb://localhost')

db = client[db_name]



# Get difference in ids

ids_new = get_ids(db, col_new)

ids_original = get_ids(db, col_original)

ids_diff = list(set(ids_new).difference(ids_original))



# Get all documents with userid that are in col_new, but not in

# col_original. Hence, all duplicates are skipped.

db[col_new].aggregate([

    {'$match': {

        field: {'$in': ids_diff}}},

    {'$out': 'col_diff'}])



# Use mongodump to save col_diff

subprocess.check_output(['mongodump',

                         '-d',

                         db_name,

                         '-c',

                         'col_diff'])



# Merge col_diff into col_original

subprocess.check_output(['mongorestore',

                         '-d',

                         db_name,

                         '-c',

                         col_original,

                         'dump/' + db_name + '/col_diff.bson'])

share|improve this answer

answered Dec 12 '18 at 12:41

WuffWuff

62

add a comment | 

0

There might be more elegent solutions for this, but let me answer how I merged a new collection into an original collection without importing the duplicates.

1. Compare userid fields in both collections and take the difference.


2. Use mongos $out operator to save documents to a difference collection. This new collection has all documents of the new collection, except of the duplicates.


3. 
   mongodump the difference collection and mongorestore it into the original collection.

from pymongo import MongoClient

import subprocess



# collection names

db_name = 'db'

col_original = 'col_original'

col_new = 'col_new'

field = 'userid'





# Get ids from collection

def get_ids(db, col):

    docs = db[col].aggregate([

        {'$project':

         {'_id': 0,

          'id': '$' + field}}

    ])

    docs = list(docs)

    ids = [x['id'] for x in docs]

    return ids





# Connect to MongoDB

client = MongoClient('mongodb://localhost')

db = client[db_name]



# Get difference in ids

ids_new = get_ids(db, col_new)

ids_original = get_ids(db, col_original)

ids_diff = list(set(ids_new).difference(ids_original))



# Get all documents with userid that are in col_new, but not in

# col_original. Hence, all duplicates are skipped.

db[col_new].aggregate([

    {'$match': {

        field: {'$in': ids_diff}}},

    {'$out': 'col_diff'}])



# Use mongodump to save col_diff

subprocess.check_output(['mongodump',

                         '-d',

                         db_name,

                         '-c',

                         'col_diff'])



# Merge col_diff into col_original

subprocess.check_output(['mongorestore',

                         '-d',

                         db_name,

                         '-c',

                         col_original,

                         'dump/' + db_name + '/col_diff.bson'])

share|improve this answer

answered Dec 12 '18 at 12:41

WuffWuff

62

add a comment | 

There might be more elegent solutions for this, but let me answer how I merged a new collection into an original collection without importing the duplicates.

1. Compare userid fields in both collections and take the difference.


2. Use mongos $out operator to save documents to a difference collection. This new collection has all documents of the new collection, except of the duplicates.


3. 
   mongodump the difference collection and mongorestore it into the original collection.

from pymongo import MongoClient

import subprocess



# collection names

db_name = 'db'

col_original = 'col_original'

col_new = 'col_new'

field = 'userid'





# Get ids from collection

def get_ids(db, col):

    docs = db[col].aggregate([

        {'$project':

         {'_id': 0,

          'id': '$' + field}}

    ])

    docs = list(docs)

    ids = [x['id'] for x in docs]

    return ids





# Connect to MongoDB

client = MongoClient('mongodb://localhost')

db = client[db_name]



# Get difference in ids

ids_new = get_ids(db, col_new)

ids_original = get_ids(db, col_original)

ids_diff = list(set(ids_new).difference(ids_original))



# Get all documents with userid that are in col_new, but not in

# col_original. Hence, all duplicates are skipped.

db[col_new].aggregate([

    {'$match': {

        field: {'$in': ids_diff}}},

    {'$out': 'col_diff'}])



# Use mongodump to save col_diff

subprocess.check_output(['mongodump',

                         '-d',

                         db_name,

                         '-c',

                         'col_diff'])



# Merge col_diff into col_original

subprocess.check_output(['mongorestore',

                         '-d',

                         db_name,

                         '-c',

                         col_original,

                         'dump/' + db_name + '/col_diff.bson'])

share|improve this answer

answered Dec 12 '18 at 12:41

WuffWuff

62

from pymongo import MongoClient

import subprocess



# collection names

db_name = 'db'

col_original = 'col_original'

col_new = 'col_new'

field = 'userid'





# Get ids from collection

def get_ids(db, col):

    docs = db[col].aggregate([

        {'$project':

         {'_id': 0,

          'id': '$' + field}}

    ])

    docs = list(docs)

    ids = [x['id'] for x in docs]

    return ids





# Connect to MongoDB

client = MongoClient('mongodb://localhost')

db = client[db_name]



# Get difference in ids

ids_new = get_ids(db, col_new)

ids_original = get_ids(db, col_original)

ids_diff = list(set(ids_new).difference(ids_original))



# Get all documents with userid that are in col_new, but not in

# col_original. Hence, all duplicates are skipped.

db[col_new].aggregate([

    {'$match': {

        field: {'$in': ids_diff}}},

    {'$out': 'col_diff'}])



# Use mongodump to save col_diff

subprocess.check_output(['mongodump',

                         '-d',

                         db_name,

                         '-c',

                         'col_diff'])



# Merge col_diff into col_original

subprocess.check_output(['mongorestore',

                         '-d',

                         db_name,

                         '-c',

                         col_original,

                         'dump/' + db_name + '/col_diff.bson'])

share|improve this answer

answered Dec 12 '18 at 12:41

WuffWuff

62

answered Dec 12 '18 at 12:41

WuffWuff

62

answered Dec 12 '18 at 12:41

WuffWuff

62