• 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
BIG MESS: Almost all songs seemed to be duplicated by Batch Import
#11
One thing I just ran into during tests was the fact that Windows is not case sensitive when it comes to file names, but Android is. So this created some scenarios where the logic failed due to Android having stored the file with lower case but Windows asked it to find the file using upper case. It looks like my logic needs to be updated to properly handle this.

UPDATE:
This is going to get messy. Different parts of the Android file system can be case sensitive or case insensitive. I won't really have a way of knowing beforehand. This creates some complexity in my logic when it comes to determining if any songs use a given file path.
Reply
#12
Drag-and-drop to Add a file works, and detects if the file already exists. But before it existed on the tablet, the very same file when included in Batch Import was being skipped.

Case sensitive doesn't seem to be a factor in my specific files, unless somehow the case got changed during import which then conflicted with a later attempt. All my files are proper cased since they are the actual song titles I want in the database. All naming and manipulation happens in Windows.

Should I try drag-and-drop of my entire set of files (close to 5000), maybe tried in batches? I don't know which files are missing, that alone would be a huge job, though I'd start with the "A..." songs since I know some of them are missing. I'll end up with many many messages that files already exist, but doing this might eventually get all my currently-lost files added. Of course that wouldn't seem to "fix" the issue with Batch Import that I'll need for the next fakebook, etc.

New clue, maybe:

I'm definitely missing songs in the top of the alphabet, down to "An..." But another "An..." imported, so likely the exact letters aren't the problem. Looking further, in Explorer that's my first 252 or so files. However, MSPC shows them in different order, dropping leading articles and certain punctuation, so I don't really know how whether the skipped files are a significant number that would be a strong clue. .But maybe Batch Import is skipping approximately the first 250 files, could be an additional 50 or more that have leading articles/punctuation.

Or is it by total list size? The entire folder I'm trying to Batch Import (skipping dupes) is 4658 + 78 in my one subfolder (XMAS songs), total 4736. Maybe the Import only handles 4500 files (not counting subfolder)? Or could it be affects by file/title lengths? (I could try Batch Import omitting the subfolder if you wish).

I see a file counter zip by when Batch Importing, could that have an affect or provide clues?

Also, can't prove this but I'm getting the sense that certain songs have been missing from MSP even before this. I keep everything in my Windows TAB folder, and I'm spotting random songs that are there but not on my tablet. Didn't notice so I don't know when or why or how it was overlooked. Could be my error, or could have been skipped by prior Batch Import.

By the way, the reason I'm doing what I'm doing is my system for managing sheet music PDFs. I have many thousands, 5 times more than I'm pulling into MSP. Each scanned fakebook/music book is in its own folder, named with the ID of the source (Ultimate FakeBook is UFB, etc). I find the best version(s) of each song that I care to play and copy it into my TAB (for Tablet) folder). Since all my files have the source ID at the end of the filename, I can manage this easily. When I find a better version, I put it in TAB and remove the old version.

With folder TAB on my PC as my master, I simply need to copy (Add or Import) the entire folder into MSPC and thereby into MSP on my tablet. Then periodically update, I thought via Batch Import with Avoid Duplicates, when I add more new songs to TAB, or update by adding/deleting. I really just need to have MSP match TAB. (This is a common professional method, nothing I invented, with bands/orchestras/shows etc where they maintain a master library.)

In my situation (and others who do the same method), I would love a way to batch import/refresh/sync the entire MSP with the Windows folder. Some control over this would be needed.

Option: If a PC file does not exist on tablet, import it absolutely (what I'm trying to do). Bypass any checking other than the OS limitation of non-duplicate path/file name. Something like "Copy file to tablet even if the song already exists with same or different title". Right now, only manually Adding seems to do this with the Batch-skipped songs. Respect that the file name can't already exist, but make that the only constraint. (And/or, give an option to auto-rename the old or new file, appending a counter for instance.)

But also, help deal with files on the tablet that are no longer on the PC (in the specified folder) because right now that's a manual process. (Currently, when I find a song on tablet that I don't want, I put the song in a special collection then later review that collection vs. what is on the PC's TAB folder. I don't know that MSPC should delete files from the PC, but it would be really helpful to flag them, or present a Only-On-Tablet list. Maybe stick a value in a standard or Custom field so all such files can be found.

PS: A handy MSPC feature would a column that shows the full path of each file, to debug goofy and confusing situations.

Sorry to be dumping so much detail, just hoping something gives you a "a-ha".
-- John

Samsung Galaxy Tab Pro 12.2 (SM-T900) Android 5.1.1
Reply
#13
There should be no limit to the number of files you can batch import. I've been working on fixes for this throughout the day, so I'll want you to try the batch import again after I release the next couple updates and we can see if it all works better for you. We can think about future changes after the current features work reliably for you.

Thanks,
Mike
Reply
#14
Please let me know when there's something to test.
-- John

Samsung Galaxy Tab Pro 12.2 (SM-T900) Android 5.1.1
Reply
#15
UPDATE: After applying new MSP and MSPC updates to tablet and Windows, I tried the Import again. Starting library had 4211 songs. Originally 648 would be imported, should have been more. Restoring tablet backup, then trying again just now, MSPC imported 764. When the process said it a file already existed on tablet, I told it to Retain Original, and checked the box to always do this.

764 was good, but not great. Several songs still did not Import. Looking directly at tablet storage, I find at least some of the files already exist, they just don't have database records. How this happened is a mystery. Whether this explains all the failures is a mystery.

Given that import recognizes that a file already exists, could it also recognize that there's no database record for it? Then either (re)create the record, or at least identify the file name so I could further deal with it. Because it would be a huge job to determine which files are orphans.

Find Missing Files didn't help, of course. Though it found some MP3 files, don't know what it thinks is missing, but I deleted them.

Running Fix File Paths found nothing, not surprising because I have always stored all sheet music PDFs in a single folder.

What else can I do? What I'm trying right now is another Batch Import. I have to check Avoid Duplicate Songs or thousand would be re-imported, I presume. Instead, when it found the first duplicate file and asked me what to do, instead of choosing Use Original File I chose Overwrite, hoping this would recreate the missing db record. (Of course, the action dialog shows the name of the file, but it is not one that seems to be missing, but perhaps other such messages would be...??)

Import just finished, and this time says it imported 766, for a total of 4213. (Prior attempt resulted in 4213). I don't know which 2 files it just added, but the missing records I'm monitoring are still unchanged, files are on tablet but records not in database.

Seems like there needs to be a more brute force repair options, some way to assures that everything exists everywhere:
-- Add Every File on PC to Tablet, skip only if already exists (perhaps this action already happens) AND make sure a database record exists for every file being added.
And/or this repair action:
-- Check Every File for a Database Record and Create Any Missing Records. Likely slow but could be a powerful cure for a messed-up database.

Would an attempt at this be to run Batch Import with Avoid Duplicates not checked, will likely take hours, but would that be worthwhile? My big hope is that I don't wipe out my annotations, collections and setlists. Any re-loaded files would be identical in filename if that helps. What does the Import process do to annotations, lists, etc when the underlying file is replaced with same filename?
-- John

Samsung Galaxy Tab Pro 12.2 (SM-T900) Android 5.1.1
Reply
#16
I'm a little surprised by your results. I added code for the batch import so that it wouldn't even perform the duplicate file check if there wasn't a song in the library already using that file. That means if you are importing new files through the batch import, one of two things should happen:

1) If a file already resides at the output path for the file (depends upon your storage settings and storage directory), then you should get prompted to handle the conflict. You absolutely do not want to have "Update songs if matching files are found" checked. If you do that, it's going to try to update a non-existent song. This is a bug in the companion app - I just added a fix so that it won't try to do this anymore, but in the meantime, don't check that option. Instead, make sure you handle the conflict by overwriting the file, and check "Apply to All".
2) If a file does not reside at the output path, the file should be imported without any issues.

You definitely want to leave "Avoid Duplicate Songs" checked, or it will just add tons of duplicate songs in your library, which you don't want. If you don't have "Update songs if matching files are found", then I'm not sure why the batch import wouldn't import all of the files like you are expecting. Can you take a screenshot of the settings in the batch import dialog? I just want to check them. Also, what is your storage location on the tablet?

Mike
Reply
#17
MSP is set to Let MobileSheets Manage My Files, which are stored here, same location since MSP installed some time ago:
/storage/extSdCard/Android/data/com.zubersoft.mobilesheetspro/files

I use flat storage, no subdirectories for songs.

The problem, possibly existing for months or longer, is that some songs I add to tablet library via MSPC do not actually get added. I've had "I thought I had that song" moments but figured I was forgetting. I can add individual songs, no problem. But now, I think there's a Batch Import problem, because that is how I've added the vast majority of songs. I have a mix of missing songs and missing files, and can't seem to correct this via Batch Import or any other MSP action.

Using the latest updates of MSP and MSPC, with Batch Import always set to Avoid Duplicate Songs, the problem is that the two forms of mismatch do not get corrected. When a duplicate file is found, selecting Use Original File, or Overwrite, or Skip File doesn't seem to matter, at least not in my test records/files. (If I'm incorrect about this, please advise what tests to run.)

I find instances of two types of database-file mismatches, and they persist, can't find a way to have Batch Import "fix" them:

Source of files is always the same, a single folder on PC that has all the PDFs I want on the tablet. Every file is precisely named, and MSPC is set to use the Filename as the Song name. For Batch Import I simply select this folder, and MSPC seems to process all the files in the folder.

When...
-Song already in database
-File not on tablet
Result=The missing file does not get added to the bad record.
Instead, the missing file gets added as a new record, so the song name exists twice in the database. In one test, the prior file-less database record was then found to link to the newly-added PDF file. Presumably this was because the db record had originally pointed at the file, which disappeared somehow but the pointer was still there.
Note that the song name that gets duplicated is identical to what was already in the library, because it is the Filename. There seems to be no checking for an existing db record of the same name.
The problem is that duplicate records appear and then must be manually spotted and deleted. The risk is if the two (or more) records do not all point at the file, probably they do, but the only verification is to open each song.

When...
-Song not in database
-File already on tablet
Result=Nothing happens with Batch Import, the song does not get added to the database, unless....
The orphan file is identified during import via a Duplicate filename dialog. If I select Overwrite, the song gets added to the database and presumably the file gets replaced. However, with more than 4000 files the only practical method is selecting Duplicate file Overwrite to do this with all files by checking the box, which results in almost the entire library being re-copied from PC to tablet, just to get any "lost" songs added back to the database. This would take a while, but is it safe? Any other side effects?
The orphan PDF file does not show up via Find Missing Files, not sure what it actually does, though in a previous test it found orphan mp3 files.

I'm trying another Batch Import, specifying Overwrite for all files. Hoping this does not result in the topic that started this post, double records for almost every song, yet some songs skipped. Here goes...

NOPE, FAILED. Importing with Overwrite of all existing files, the library on tablet grew by ONE file. The Library still is missing more than 500 files in the PC's source folder total of 4658 files. The tablet folder that stores MSP songs has about 4940 files, apparently are on the tablet but not in the database therefore unidentified by MSP.

This is a mystery. I have never deleted a song without also deleting its file (except for a few recent tests), so how did these files get on the tablet and/or their database records get deleted? Does MSP copy new files from PC to tablet, THEN update the database, which perhaps failed for some reason? It will be difficult to delete them because they look just like every other MSP library file. This is where the solution could be for MSP having a mode where it scans all files for matchng database records.

The Proper outcome would be for ALL files from the source get loaded onto the tablet, no more or less. I see no cause for these large, persistent anomalies. Or am I thinking about this wrong?
-- John

Samsung Galaxy Tab Pro 12.2 (SM-T900) Android 5.1.1
Reply
#18
I'll run some more tests to see if I can reproduce what you are seeing now that I know your setup. MobileSheetsPro does send the files first, then sends updates to the database. The all happens after the batch import steps through every file though. If you caused the connection to fail in the middle of sending the files, then you could end up with a situation where files are left on the tablet without database entries. I suppose this is worse than having an updated database where you have all the songs, but not all the files. I can switch this around without too much difficulty.

Selecting "overwrite" for all duplicate files is fine for what you want to do. I'm not sure why it didn't work for you though. I'll have to see what happens if I create some tests to replicate that scenario.

If there are bugs preventing this from working, I'll get them fixed. In the tests I performed earlier, I was able to batch import thousands of files without any problems. What I need to do next is create a scenario where I import a lot of files, then add a ton of files to that directory, and import it again to make sure all the new files are correctly imported. If it all succeeds, then I'll have to go further by trying to create more scenarios where conflicts will occur. I appreciate your patience while we get this all figured out.

Mike
Reply
#19
Mike, a few more facts about the files in MSP's storage location on tablet:

I have confirmed in 3 ways that there are 4941 PDF files on tablet, though the library contains 4222.

My source folder on PC has 4736 files, comprised of 4658 in top level folder plus 78 in one subfolder (used for XMAS songs). Batch Import of this folder does not add more to the tablet.

Looking on the tablet at Date Created of the PDFs, I see a few chunks that match the dates I tried additional Batch Import and tests, but the oldest date is June 15, 9 days ago. If that is accurate, the problem of file- not-in-database happened at least that recently, but probably due to me restoring a backup, which I did around that date or slightly later. The MSP on-tablet Restore process recommends and I always agree to wipe the existing files, then reload them from the backup. If the storage emptying is 100%, the "lost" files got restored from the backup, along with the database that did not have records for all of the files.

Trying to determine when the "extra" (and "lost") files showed up, I just Restored a tablet backup from Jan 24, to see how many files. Result: 4211 files were restored, which matches my current MSP Songs total of 4222 (one extra due to experimenting). Unfortunately, the date of all the restored files is today. But looking at the storage folder, there are 4930 PDF files, even more than before restoring Jan 24 backup. This might make sense because in-between I did some clean-up, removing from the tablet various PDFs that were poor scans, unwanted songs, etc. But of course, the files count as of Jan 24 also greatly exceeds the database count, so apparently whatever sent MSP off the tracks happened months (or years?) before.

Unfortunately, I don't have an older on-tablet backup, Jan 24 is the max of my Way-Back Machine.

But ... I found an older backup yet don't know an effective way to use it for investigation. On my PC I have a older MSPC backup from May 19, 2015. MSPC doesn't offer to Restore this file, so I opened it in View mode to see whats what. Viewing this file in MSPC doesn't give me a count of records nor does it show file dates. I don't know the diff of EXtract Database, Extract, and Extract All, so I tried Extract Database, which dumps out a copy of the .db file.

Then I tried Extract All, which copies all the song files from the backup onto my PC's drive, but resulted in many "Error crating image file", the "Error creating:" seems to be showing Placeholder records, which don't have physical files and/or the Placeholder title text would not be a valid filename. There's no Skip Similar option so I had to work through quite a few of these. (I use Placeholders mainly to have song titles that my band plays from memory, to put them in Setlists, and to remind me to keep looking for or create an actual chart.)

Extract All resulted in a separate folder for each song (which is not how I have set MSP, I don't see a benefit)). Assuming each folder has one PDF, that's 3851, which simply indicates that I added hundreds more songs since May 2016. Folders and files are all dated today. But since I can't seem to Restore this MSPC backup, I can only view and extract it, I don't know how many database records exist.
-- John

Samsung Galaxy Tab Pro 12.2 (SM-T900) Android 5.1.1
Reply
#20
I am running through more complicated tests, and I'm definitely encountering some problems similar to what you've described. I'm going to keep working through the issues, and I'll let you know when an update is available.

Mike
Reply


Digg   Delicious   Reddit   Facebook   Twitter   StumbleUpon  


Users browsing this thread:
1 Guest(s)


  Theme © 2014 iAndrew  
Powered By MyBB, © 2002-2018 MyBB Group.