Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix/160 invalid date format on import csv #7525

Merged
merged 15 commits into from
Dec 16, 2024
Merged
Show file tree
Hide file tree
Changes from 10 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion app/api/csv/csvLoader.ts
Original file line number Diff line number Diff line change
Expand Up @@ -132,7 +132,8 @@ export class CSVLoader extends EventEmitter {
template,
file,
propNameToThesauriId,
indexedTranslations
indexedTranslations,
dateFormat
);
this.emit('entityLoaded', entity);
}
Expand Down
56 changes: 29 additions & 27 deletions app/api/csv/importEntity.ts
Original file line number Diff line number Diff line change
@@ -1,8 +1,10 @@
/* eslint-disable no-await-in-loop */
/* eslint-disable no-unreachable-loop */
/* eslint-disable no-restricted-syntax */
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We really cannot put exceptions to es-lints like this. The fact that this practice has happened before should not be followed these days. So, please, remove this exceptions and lets fix the code to not have these eslint warnings:
To solve this, please create a reduce with an async callback.

Also, the no-unreachable-loop is not really solving any warning, so this was probably a previous version of the code which is no longer present, all the more reason not to include this exceptions that can later be left in the header of the file and we end up with code that is against our own guidelines.

// eslint-disable-next-line node/no-restricted-import
import { createReadStream } from 'fs';
import entities from 'api/entities';
import { search } from 'api/search';
import entitiesModel from 'api/entities/entitiesModel';
import { processDocument } from 'api/files/processDocument';
import { RawEntity } from 'api/csv/entityRow';
import { TemplateSchema } from 'shared/types/templateType';
Expand Down Expand Up @@ -174,34 +176,34 @@ const translateEntity = async (
template: TemplateSchema,
importFile: ImportFile,
propNameToThesauriId: Record<string, string>,
indexedTranslations: FullyIndexedTranslations
indexedTranslations: FullyIndexedTranslations,
dateFormat?: string
) => {
await entitiesModel.saveMultiple(
await Promise.all(
translations.map(async translatedEntity => {
const translatedEntityObject = await entityObject(
{
...translatedEntity,
propertiesFromColumns: {
...translatedEntity.propertiesFromColumns,
id: ensure(entity.sharedId),
},
},
template,
{
language: translatedEntity.language,
}
);
for (const translatedEntity of translations) {
const entityParsed = await entityObject(
{
...translatedEntity,
propertiesFromColumns: {
...translatedEntity.propertiesFromColumns,
id: ensure(entity.sharedId),
},
},
template,
{
language: translatedEntity.language,
dateFormat,
}
);

return translateSelectLabels(
translatedEntityObject,
translatedEntity.language,
indexedTranslations,
propNameToThesauriId
);
})
)
);
const toSave = translateSelectLabels(
entityParsed,
translatedEntity.language,
indexedTranslations,
propNameToThesauriId
);

await entities.save(toSave, { language: translatedEntity.language, user: {} });
}

await Promise.all(
translations.map(async translatedEntity => {
Expand Down
59 changes: 58 additions & 1 deletion app/api/csv/specs/csvLoader.spec.js
Original file line number Diff line number Diff line change
@@ -1,13 +1,15 @@
/* eslint-disable max-statements */
/* eslint-disable max-lines */
import path from 'path';

import { CSVLoader } from 'api/csv';
import { templateWithGeneratedTitle } from 'api/csv/specs/csvLoaderFixtures';
import { simpleTemplateId, templateWithGeneratedTitle } from 'api/csv/specs/csvLoaderFixtures';
import entities from 'api/entities';
import translations from 'api/i18n';
import { search } from 'api/search';
import settings from 'api/settings';
import db from 'api/utils/testing_db';
import moment from 'moment';
import typeParsers from '../typeParsers';
import fixtures, { template1Id } from './csvLoaderFixtures';
import { mockCsvFileReadStream } from './helpers';
Expand Down Expand Up @@ -400,4 +402,59 @@ describe('csvLoader', () => {
});
});
});

describe('should parse date respecting the dateFormat on settings collection ', () => {
beforeEach(() => jest.restoreAllMocks());

const setDateFormat = async dateFormat => {
const _fixtures = { ...fixtures };
_fixtures.settings = [
{
..._fixtures.settings[0],
languages: [
{ key: 'en', label: 'English', default: true },
{ key: 'es', label: 'Spanish' },
],
dateFormat,
},
];
await db.setupFixturesAndContext(_fixtures);
};

it('should correctly parse MM/dd/yyyy', async () => {
const dateFormat = 'MM/dd/yyyy';
await setDateFormat(dateFormat);

const dateOnCSV = '12/31/2024';
const csv = path.join(__dirname, '/simple_template.csv');
const selectedLanguageOnUserInterface = 'es';
const expectedDate = moment.utc(dateOnCSV, [dateFormat.toUpperCase()]).unix();

await loader.load(csv, simpleTemplateId, { language: selectedLanguageOnUserInterface });

const [englishEntity] = await entities.get({ language: 'en' });
const [spanishEntity] = await entities.get({ language: 'es' });

expect(spanishEntity.metadata.date_field).toEqual([{ value: expectedDate }]);
expect(englishEntity.metadata.date_field).toEqual([{ value: expectedDate }]);
});

it('should correctly parse yyyy/MM/dd', async () => {
const dateFormat = 'yyyy/MM/dd';
await setDateFormat(dateFormat);

const dateOnCSV = '2024/12/31';
const csv = path.join(__dirname, '/simple_template_2.csv');
const selectedLanguageOnUserInterface = 'es';
const expectedDate = moment.utc(dateOnCSV, [dateFormat.toUpperCase()]).unix();

await loader.load(csv, simpleTemplateId, { language: selectedLanguageOnUserInterface });

const [englishEntity] = await entities.get({ language: 'en' });
const [spanishEntity] = await entities.get({ language: 'es' });

expect(spanishEntity.metadata.date_field).toEqual([{ value: expectedDate }]);
expect(englishEntity.metadata.date_field).toEqual([{ value: expectedDate }]);
});
});
});
21 changes: 20 additions & 1 deletion app/api/csv/specs/csvLoaderFixtures.ts
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@ import { TranslationDBO } from 'api/i18n.v2/schemas/TranslationDBO';
import { getFixturesFactory } from 'api/utils/fixturesFactory';

const template1Id = db.id();
const simpleTemplateId = db.id();
const multiSelectThesaurusId = db.id();
const thesauri1Id = db.id();
const templateToRelateId = db.id();
Expand Down Expand Up @@ -48,6 +49,24 @@ const commonTranslationsV2 = (language: LanguageISO6391): TranslationDBO[] => [

export default {
templates: [
{
_id: simpleTemplateId,
name: 'Simple template',
properties: [
{
_id: db.id(),
type: propertyTypes.text,
label: 'simple text field',
name: templateUtils.safeName('simple text field'),
},
{
_id: db.id(),
type: propertyTypes.date,
label: 'Date field',
name: templateUtils.safeName('Date field'),
},
],
},
{
_id: templateToRelateId,
name: 'template to relate',
Expand Down Expand Up @@ -180,4 +199,4 @@ export default {
],
};

export { template1Id, templateWithGeneratedTitle, thesauri1Id };
export { template1Id, templateWithGeneratedTitle, thesauri1Id, simpleTemplateId };
2 changes: 2 additions & 0 deletions app/api/csv/specs/simple_template.csv
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
Title,Date field,simple text field__en,simple text field__es
Simple template title,12/31/2024,English,Spanish
2 changes: 2 additions & 0 deletions app/api/csv/specs/simple_template_2.csv
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
Title,Date field,simple text field__en,simple text field__es
Simple template title,2024/12/31,English,Spanish
Loading