Too big project? (exclamation marks & fatal errors)

Absolute Pyramid lover here. I have 150k great things to say about this machine. Today’s post unfortunately is a cry for help. I am aware support is deprecated… I have the latest OS installed.

I’m working on an ambitious project (~15 tracks with 20 patterns each to run through 20 sequences). Time signature is 3/4, track length 18. Fun stuff.
Until I got a frozen screen, when I (accidentally?) tried to rotate all notes within a pattern. We are talking 4 note-chords at most, but there are ~50 of these in a pattern. I panicked a bit and shut down the device.

After another couple of days I suddenly got an Error message with a couple of number codes, which I can’t remember (I thought this error would reappear, but it never did… and I don’t believe those are logged in the device or SD Card afaik). 5 min later, I got a Fatal Error (frightening experience). Since then I work in fail-soft mode, with a lot of care, I do minimal moves, I don’t use the rotate function anymore, I copy/paste small amount of notes etc. workflow is degraded but progress is possible.

Today I started getting Fatal Errors erratically when I would change Sequence Number. Thought it was time for an SD formatting…

I have formatted my 16Go SD card (long formatting), reinstalled the OS, and put back my project in there. No Fatal Errors yet, but memory is playing new tricks…

Sometimes I get an exclamation mark (out of memory, if I’m not mistaken) upon loading the project, sometimes I don’t.
Copying a track is simply not responding. Copying a pattern of a track is not responding neither. Copying all notes within the pattern of a track sometimes works, sometimes doesn’t, and sometimes gives me another exclamation mark.
Sometimes I can copy successfully, but it’s when I paste that I get the exclamation mark (only some notes are pasted). What is weird is that I have managed to reproduce scoping down to getting an exclamation mark for trying to copy a simple, individual, 4-note chord.

Two questions:

  1. Is my project too big / does anyone work with the Pyramid on even bigger projects?

  2. Is it possible to have a corrupted project ? Not SD Card, not device, but actual data in the project? (I’d like to know if I should prepare a backup plan to finish my project from my laptop for instance)

I don’t know how to help you, but 8 days without out a reply from Squarp is messed up. I hope you sent a ticket or email. Good luck!

1 Like

That does sound like an ambitious project!

If the 15 tracks have 20 patterns each and the tracks and patterns are all 18 bars long then, I think that would mean, each bar would have to have less than 2.4 MIDI events in it on average to keep within the stated limit (or less than an average of 43.3 events in a pattern). So I think it is quite likely that you may have reached that limit.

Some of the inconsistences you describe do seem a bit odd. I have no idea how the memory in Pyramid works, though. Perhaps there’s some allocated (reserved?) RAM for the copy buffer, perhaps other RAM can be used if it’s available but maybe that depends on something like how fragmented it all is.

I’m afraid I have no idea whether it is possible to have a corrupted project.

2 Likes

Hi @camzeech,

You say each track in your project has 20 patterns, and that these are all 18 bars long? Is that the case for each and every track? Obviously, i don’t know what your project is like but I can’t help wondering if you really need a new pattern for every track for each sequence change or whether all really need to be 18 bars long.

I have found myself tending towards longer track/patterns (maybe a bit like you) as I’ve got to know Pyramid (and the sorts of things I want it to do) a little better. This puts less strain on the limited number of ‘sequences’ you have with Pyramid, and changing them at less frequent intervals with regular sized (and longer) patterns means you can get some complex (and possibly evolving) arrangements going on. But, unfortunately, it might not be all that efficient with respects to the number of MIDI events your project will have to store in its relatively small memory. It might be that you end up with a fair amount of duplication of MIDI events across your patterns because you want some (small?) variations at some specific places in your project and there aren’t really enough (Pyramid) sequences available to use a sequence for every small variation you might want to make. So it might seem that you are a bit stuck if you want to make long complex arrangements with your Pyramid.

But I don’t think that is necessarily so. It might be that all the tracks in your project are ever evolving and that there is next to no repetition of any bars or phrases. If that is so there may be little that can be done. But presuming there is at least some repettion of bars or phrases in your project I think there probably are ways to reduce the amount of memory you are using with this project. Although that really will depend on what is going on in your project and where you want it to go.

A couple of rhetorical questions: Could you have made some of the (later) sequences you have there from any of the patterns that already existed in other sequences? Are there some patterns in a sequence that you have that could be made by switching from one sequence to another at some point within the 18 bars? You have 12 more sequences available and they are relatively memory cheap. So, for instance, you could create a new sequence to change one (or more) patterns over at the appropriate point during the 18 bars. Whether this would work will, of course, depend on what run modes (and possibly other settings) you are using (it will also mean another sequence and a little more messy chain of sequences). Thinking about that, playing with when sequences change and using patterns with different run modes can be a good way to create some variation from largely pre-existing sequences. But I suppose that would be of more use at an earlier stage than you are at currently with this project in question. It requires creating new patterns (if you need to keep the old ones with the previous run mode), though if I understand correctly from your description of your project, your existing patterns would behave the same whether they were in relatch mode or free mode because it seems your sequence changes come at regular intervals, every 18 bars and the patterns are (all?) 18 bars long (which would probably mean the trigger mode would also behave the same way).

Personally, when I have a few relatively long sequences arranged I tend to start looking for ways to create another (probably of the same length) by looking for places to switch between the sequences that I already have. I might have to create a few new patterns for some tracks to get the results I want, so I will probably be creating another sequence or two as well, but I’m unlikley to need to create new patterns for all the tracks. So maybe I might get another half minute or so arranged at relatively little cost (memory-wise), and I’ll have more sequences available to play with to help create ‘another’ future sequence. And who knows? I may need to create fewer new patterns to get something I like this time. It’s sort of like the opposite of diminishing returns, maybe. Of course, sometimes I’ll need to make something more like a completely new sequence with nearly a full set of new patterns, it really does depend. But I do like to try and recycle. But that’s me and one way I like to work with my Pyramid. What works for me might not for other people. One of the good things about Pyramid (and probably many other sequencers) is that it allows for many different approaches. However there are constraints, and Pyramid’s style of arranging (sequence mode) and its fairly small amount of memory are probably the most restrictive of these when it comes to making long, complex arrangements (at least those that have a fixed and repeatable order). The challenge, of course, is to find ways of doing things that allow you to get the sort of results you want within those constraints.

I must admit that I am looking at what you have said and seeing the words, ‘~15 tracks with 20 patterns each to run through 20 sequences’, and thinking that that sounds a bit likely to be a bit memory intensive and I can’t help thinking that maybe you may have some (unnecessarily) duplicated patterns amongst all that lot. I may have misunderstood or taken what you said (that would be the ‘each’, I suppose) too literally and I may be way off mark with what I’m saying and what I’m about to say. Apologies if that’s the case.

I wonder if perhaps you use the auto pattern feature, and keep it on (set to ‘at new seq’ in the misc settings). I could see this leading to a bit of unnecessary duplication of MIDI events, because in wanting to keep the patterns chaging in tandem with their matching sequence (which might help you keep track of where you are/what you might be editting) you might end up copying much, or all, of the content from another pattern into the new (empty but ‘correctly’ numbered) one. Now, you probably don’t keep auto pattern on and you are thinking that I’m a right patronising git with nothing helpful to say. Or you may be thinking, ‘I like that feature and I don’t make unnecessary duplicates’. And, to be honest, I have wondered whether I ought to mention it. But in the end I found myself wondering about those ‘20 patterns’ and ‘20 sequences’ and couldn’t help thinking maybe you did have auto pattern on and that it might be part of the problem. I think that whilst it can be helpful near the start of your tune-making it may not be later on. The thing is I don’t know why (or even really, if,) you have 20 patterns for each and every track and if the 20 sequences each change a pattern to the corresponding sequence number, or whether it’s just a coincidence that there’s 20 of each. So I probably should just say nothing, but then again, what if me saying something actually helps? I could have just asked but I felt that it might require an answer (and another reply) before we got anywhere. So I decided to risk coming across as a bit patronising. Apologies if I have, I do hope I haven’t.

1 Like

I don’t believe it’s messed up at all. I have not written to Squarp and do not intend to. Pyramid’s support is over: their support time is more valuable on their newer products and I respect that.

I think you nailed it. I have spent a couple of hours doing a lot of testing around on copied versions of this project, and I found the following: if I delete, say, 6 heavily loaded tracks from my project, I can then, from the remaining ones fully copy 1 (heavy) track with its 20 patterns and all its notes again. If I don’t delete the 6 tracks, it won’t let me copy fully that track.
So yes, copy/paste clipboard memory seems impacted by what Pyramid is supposed to cope with elsewhere (loaded tracks existing in project). So there’s a chance data corruption is not on the table (although to be precise, I still need to validate that those 6 tracks are not coincidentally the source of some other bug).

Obviously, i don’t know what your project is like but I can’t help wondering if you really need a new pattern for every track for each sequence change or whether all really need to be 18 bars long.

I’m working on getting Ravel’s Bolero fully electronic (lulz) so yea each bar, pattern, track is different (even velocity-wise).

All your questions were great:

  • yes I have auto-pattern on,
  • but no, I don’t leverage all patterns: I have deleted any pattern where an instrument is not supposed to play. Imagine the flute plays sequences 1-5, then patterns 6 and up are all empty: I am assuming an empty pattern doesn’t consume memory, even if the sequencer switches to its number during play mode.
  • What I did not do is do the extra mile and try to leverage a given pattern several times if it happened to be identical, but I think I might get at best 10% off my project load. You make me realize I could delete some patterns for some instruments (the snare namely) and boil them down to say 5 instead of 20, but I can’t do this for more complex instruments otherwise I would need a written mapping of which patterns should play when (which is super hard to manage when there are 10+ instruments)
  • What I hope, however, is that merging tracks together could ease memory (same amount of midi events, but on fewer tracks).

My plan now is twofold:

  • refine the research and see if deleting patterns (instead of entire tracks) relieves the clipboard too
  • do some cleaning and consolidation, maybe via computer hopefully if MIDI editing is made easier on Reaper (Kenny Gioia to the rescue !!).

I’m happy to keep anyone posted on how Track/Patterns are easy/uneasy to edit via DAW and brought back to the SD Card, if that is of interest.

Thanking this awesome community again.

Two new findings [EDIT]:

  1. Refined research shows it’s indeed not tied to amount of Tracks. If I delete a lot of patterns of a track (except one, so the Track still exists), and I repeat this operation on many tracks, I’m eventually able to copy a heavily loaded track. Memory of the entire project restricts memory of the clipboard.
    1bis) Refined testing shows it’s indeed not tied to amount of Patterns. If I delete all notes within all Patterns (except one, so the Pattern still exists), and I repeat this operation on many Patterns, I’m eventually able to copy a heavily loaded track.
    This means consolidating Patterns between each other on a DAW will not help: the only way out is to reduce the total number of midi events.

  2. An exclamation mark (kind of saying loading, or copying, or pasting, was partial), might overwrite upon save (I’m not sure anymore). Not sure if saving a project despite an exclamation mark might lead to some notes disappearing in the process…

My ask to the Squarp team :pray: (if they ever read this post), is for their future products not to raise warnings in case of memory issues (such as partial load, copy or paste), but to raise a real error. Either the content is copied, loaded, pasted successfully, or it’s not: any in between is unusable for the user imho.

Yes, I think the overall amount of MIDI events in the project is the key. The amount of tracks and patterns in themselves don’t really have any real impact on the memory (it’s what is in them that counts).

You might be able to get away with using some tracks just containing stepped CC automation (parameter locks), say, to control volume of certain channels to enable you to get more mileage out of some of the patterns that contain note information. Maybe you have some patterns that are basically copies of other patterns except they play louder. Instead of having two (or a few) 18 bar patterns you could perhaps get away with one, and a separate automation track with some patterns each with perhaps very few (as little as one?) stepped CC automation changes in them to do the volume (or some other synth parameter) change.

It might be worth seeing what you could do with some of Pyramid’s FX also.

1 Like

@camzeech, I forgot to say that I would be interested to hear about your experiences with importing and exporting MIDI files if you do end up doing that.

And I also meant to ask, if you don’t mind and if you know, what size was your project (in KB). It would be good to have an example of the amount of KB a ‘too big’ project can take. If you could give the total of the project folder and perhaps the how big your core.pyr file is as well. The 13000 MIDI event limit is a bit of a guide but I don’t really know how many MIDI events projects that I’ve made have contained. But I can get to see how many KBs my project folders are on my computer.

Incidentally, I notice that the core.pyr files on the very early test projects I saved were 32KB and that projects a little later on were 64KB (these are from when I backed up my SD card quite a while ago, so my core.pyr files may be quite a bit bigger than that now (which reminds me, I really ought to get and do some more backing up!)). I wonder if that difference in the core.pyr file size has more to do with the fact that I will have saved settings since those early tests or whether it has more to do with having done stuff like enabling patterns, used more tracks, etc.

I did try dropping a few MIDI files into Ableton (Lite) and for each MIDI file it automatically put the patterns on separate MIDI tracks (I don’t really use Ableton other than to record my mixes, so I don’t really know what I’m doing). So I guess the MIDI file contains some information about patterns. It seemed to take into account the length of the patterns and it also asked whether I wanted to import the tempo when I drag and dropped each MIDI file into it, so I guess tempo information is at least included with the MIDI file (though I doubt each track needs that stored individually in Pyramid’s memory). I suppose as each track/pattern can have its own time signature so there may be some (Pyramid) memory used for that per track/pattern (though that may be stored in the ‘core’ rather than the RAM (if those are even different things).

I did have a track that was just one note in one pattern (it was probably used just to trigger a switch in my modular synth). The MIDI file for that was 81 bytes. Another MIDI file with about 35 notes in was 728 bytes. So I suppose that would suggest that tracks/patterns do have a bit of a cost, but that’s a MIDI file and we don’t really know if all of what is in a MIDI file takes up space in the (same) RAM/memory that we can use for MIDI events in our projects. I did also have some MIDI files containing purely CC data. Around 50 changes in CC value took around 600 bytes (in half the number of patterns and a bar less than half the overall length of the MIDI file compared to the 35 note, 728 bytes track).

I suppose Squarp talk about MIDI events in respect of project memory for a reason and looking at the bytes in a project folder might not give that good an impression. Still, I think I would be interested to know what the size of your Bolero project folder was when it approached the limit.

I was sure I read on this forum somewhere that there is some way of seeing how much memory you have remaining on Pyramid (I guess it might be a percentage, which would be sensible and handy). If someone knows if you can, and if so how, it would save me trawling through some old topics on here or through the manual in a more systemmatic way than I already have (I am less sure that I read that now). EDIT: A-ha! It looks like it’s probably in the obvious place, the Info screen (sub-menu?), in the Settings Menu. At least the Quickstart guide in the manual says ‘memory’ is one of the bits of info included there. The Settings Menu section of the manual omits it (though that includes PyrOS version as one bits of info, and the Quickstart doesn’t). I haven’t checked on an actual Pyramid yet though.

Maybe you have some patterns that are basically copies of other patterns except they play louder

Unfortunately I’m playing on velocity changes to get not only volume changes but also subtle filtering dynamics etc come into play throughout the piece. And I even got to slightly alter velocity on a note by note basis to be more realistic…

I forgot to say that I would be interested to hear about your experiences with importing and exporting MIDI files if you do end up doing that.

I’m using Reaper. I leveraged some content in other discussions / tutorials to figure that Pyramid’s midi files, when using “patterns” mode, will be imported as single file type 1 MIDI (meaning multiple tracks, by opposition to type 0, which is a single track). There are ways, from reaper, to export back multiple tracks into a single midi, it requires a couple of merging techniques and export parameters, and you want to make sure the names of the items match pattern naming in Pyramid. I haven’t been to the end of that process and reimport the files back into Pyramid, as I eventually managed to bring back my project to a usable state (the initial cleaning solution worked pretty well, I got 90% stability back!)

what size was your project (in KB). If you could give the total of the project folder and perhaps the how big your core.pyr file is as well.

Before cleaning (unstable project): core.pyr sizes 60,3 Kb (64,0 on disk). Entire project sizes 162Kb (196 on disk).
After cleaning (back to acceptable use): core.pyr sizes 51,3 Kb (56,0 on disk). Entire project sizes 125 Kb (232 on disk).

This correlates with your finding of 64Kb.

I wonder if that difference in the core.pyr file size has more to do with the fact that I will have saved settings since those early tests or whether it has more to do with having done stuff like enabling patterns, used more tracks, etc.

No idea :slight_smile: . Maybe a firmware update bumped up the limit from 32 to 64, no idea if that is even possible.

So I guess the MIDI file contains some information about patterns.

Yup, see my type 0, type 1 midi files answer above (there’s good explanations on the web). Importing tempo is standard in DAWs (Reaper offers it too) but I can’t tell if that is time signature or tempo or both really…

I haven’t checked on an actual Pyramid yet though.

Oh wow I thought this would indicate memory used wrt SD card, but you’re right it’s memory used wrt project’s limits ! Much more useful info. Funily enough I get another number here on the cleaned project: 143Kb. So it’s somewhere in between the 125-232 range that I mentioned earlier.

Super cool discussion.

It sounds like most of the control of your synth voices on this project is tied to the note information a pattern sends (note number, velocity, length, offset) and velocity, especially, is responsible for much of that. I imagine it would require quite a few CC/FX automation tracks, not to mention a fair bit more work, if you sought to divorce some of the synth parameter controls from the MIDI note events and you might very likely end up generating more MIDI events that way anyway in order to get the same sort of intentional, subtle control. Having those synth parameters tied to velocity may well be the most efficient way (in terms of total number of MIDI events) for your particular project.

I feel some frustration on your behalf because from my limited understanding of Bolero it is essentially built around a couple of repeating melodies and this should lend itself quite well to being recreated using sequencers like Pyramid, I would have thought. But because there’s a number of instuments coming in and out and the general build up in loudness, etc, it maybe isn’t so easy to make it work so well without a fair few patterns. I think Pyramid might provide some useful ways to compose a ‘Bolero type’ piece of music in a fairly MIDI event efficient way but maybe it’s not as efficient for reproducing it in a reasonably true and sympathetic type of way.

It’s good to hear you have manged to clean the project up to a level where you can work on it again. I hope you may even have enough memory available to get to point where you’re happy it is complete (or complete enough). I’m not sure what the ‘cleaning’ quite entailed. Was it mostly doing away with a few patterns (like the snare ones)?

Those figures you give for before and after cleaning are interesting. The thing that stood out initially to me was the number for the post cleaning ‘on disk’ project size. It was bigger than the pre cleaning figure despite the actual data in the project being reduced. It was also interesting that you got the core.pyr file size down quite significantly too. I was also intrigued to hear that Pyramid’s on board ‘memory used’ figure for the project was bigger than the data on disk.That is quite a surprise to me as Pyramid figures seem to always be lower for the projects I’ve compared.

From what I can gather the ‘on disk’ figure sort of refers to the amount of ‘space’ on the disk that the data is spread over. I think data is stored in ‘clusters’ on a disk and you end up with some parts (memory ‘cells’?) of the clusters not being filled (and I think, as such, that ‘space’ is no longer available). The clusters seem to need to be of a certain minimum size (?) and it seems certain data needs to go in a new cluster perhaps because some of the bytes need to run concurrently/can’t be broken up, or something. It’s good that you can still get to see the amount of memory the actual data itself takes up too, as this seems to be a whole lot closer.

I compared my old saved projects’ sizes (as seen on my computer) with my Pyramid’s information display (settings, info). Like I said, none of the sizes on Pyramid were as big as the data figures I see on my computer.

One thing I noticed was that the ‘on disk’ figures for all of my individual MIDI files was 4KB, no matter whether their data size was saying 81 bytes or 881 bytes. So I suppsoe those ‘on disk’ figures are always going to be larger than the data size figures and there will be a greater difference the more tracks they contain because each track requires at least 4 KB of space ‘on disk’.

I wonder what happens when the data in a MIDI file goes over 4KB? For instance, would 4.1 KB of data in a MIDI file require 8 KB of disk space? I suspect it might. Maybe some of your Bolero MIDI files contain more than 4 KB of data.

I think the ‘on disk’ figure is probably not too relevant for us when trying to get an impression of how much Pyramid memory (RAM) our projects use. The data size is probably better but I think Pyramid’s Information screen will be the best guide.

Having said this another thing I noticed was that the information screen on Pyramid does not always give the same figure for the same (unaltered) project. I think you only really get a ‘true’ reading if the project is the first project you have loaded since powering up.

I loaded different projects in on Pyramid to see how the on board ‘KB used’ display compared to the sizes I could see on my computer. In doing so I think I spotted that I was getting some unexpected differences with a couple of ‘template’ projects that I’d saved that were basically the same. I reloaded one and this time it appeared to have grown. I reloaded the other and apparently so too had that. I went round loading other projects and it seemed this ‘growing’ was happening for all the projects. Eventually Pyramid’s save/load function stopped working (2nd Save/load did nothing at all) though other functiuons appeared to all work still.

I powered down and after a bit powered back up. Now my projects were smaller again. I tried loading the same project (over itself). It went up by a couple of KB, for a couple of reloads, then by 3 KB, then a couple of reloads later by 4 KB. It seems loading a project from when there’s already a project in RAM does not completely clear all the RAM, at least not in a way that leaves it all available. To clear the memory completely (or as much as you can) you need to power down. Even loading a ‘NEW’ project over an existing one will be bigger than if you use load ‘NEW’ from a freshly booted and ‘empty’ Pyramid.

By the way, the 64 KB sized core.pyr files I had weren’t the ‘on disk’ figure. My computer (Macbook Pro from 2014) rounds the KB sizes to the nearest whole KB in the file list view (the MacBook’s Finder app window) and on the top left corner of it’s ‘get info’ window. The ‘on disk’ size was 66 KB. It appears that the core.pyr file size varies from project to project, though it does seem to tend towards certain ranges, in my case 32, 54, and 64 KB. I think the old 32 KB core.pyr files were made with the previous firmware, so your hunch could be right, the firmware update may be the reason why more recent ones all seem to be at least 54 KB, even my template projects. I bought a second hand Pyramid recently and the previous owner hadn’t completely cleared their SD card. Those, largely test projects, from what I can gather, had core.pyr files that were 55 KB (according to the list with Finder). I don’t know quite what influences the size of the core.pyr files but they seem to grow with the amount of tracks/patterns in use and I think that the (non-default) settings may have some affect too. I think I’d probably need to save a few different test project examples to get a better handle on this.

I suppose that whatever the core.pyr file consists of once it is saved to SD, it isn’t quite the same set of data that is stored in Pyramid’s RAM or at least that it requires less space in RAM than it does on an SD card. My ‘template’ (empty) projects use 28 KB according to Pyramid, yet their respective core.pyr files contain about 54 kB of data (the ‘on disk’ size is 57 KB). It seems that Pyramid doesn’t need to have all that core.pyr data on the SD card loaded into its RAM. I have the feeling that the MIDI file data (not ‘on disk’ figure but the total of all the data sizes for each MIDI file) would broadly take up the same amount of memory in Pyramid’s RAM. But that’s not easy to tell because some part of the core.pyr file seems to take up space in the RAM and in a project with very few tracks and MIDI events that seems to occupy most of the ‘used memory’ by a long way.

I just had a look at what the SD card contained for a more recent project. That has 8 tracks, I think only a couple have more than 1 pattern, and none have more than 3 patterns. None of the tracks are over 16 bars long. When (auto) loaded after turning on my Pyramid the Information screen displays 31 KB (how much Pyramid RAM is used). On my computer the information for the saved project (on SD card) says there is a total of 57,816 bytes (data) and there is 328 KB (!) ‘on disk’. The core.pyr file contains 55,544 bytes of data, and I totalled up the MIDI file data which came to 2272 bytes (equalling the difference of the core.pyr file subtracted from the project folder contents). Incidentally, the ‘on disk’ figure for the core.pyr file is 66 KB. Clearly though, 328 - 66 ≠ 32 (8 MIDI files at 4 KB each), it = 262 KB which suggests I have whopping MIDI files. But I don’t think I do. The biggest MIDI file is 609 bytes (of data). However, the on disk size (for each MIDI file) is 33 KB. The difference I think is because I tried naming the tracks for the first time on this project. It seems the track names get stored along with the MIDI files so they can perhaps be seen in a DAW (they do not appear on the SD card directory, but can be seen in the info my computer displays (under ‘more info’). It makes you realise just how efficient MIDI is when you can store songs worth of information in a few KBs yet it seems to take 29 KB more disk space just to have the ability to tell a DAW a track name.

What you say about MIDI file types 0 and 1 sounds right (and familiar from scraps I’ve read here and there).

This topic was automatically closed 21 days after the last reply. New replies are no longer allowed.