Gardener

The problems you mention are exactly the same ones that Schönberg, Berg, and Webern had, to various degrees, so you're in good company there. Schönberg, for example, never really could leave behind traditional form concepts and wrote atonical pieces that are more or less identical to classical sonata forms etc., even though he managed to free himself of these forms a bit more after he had started to compose with the 12-tone technique. And you'll notice that the pre-12-tone pieces of Schönberg and the early 12-tone pieces of Berg and Webern all tend to be rather short. One reason for Schönberg to start writing 12-tone music was actually that he felt that he needed such a system to be able to composer larger scale pieces as, like you, he felt it hard to keep a large scale piece together without any structural system.

Webern never managed to get beyond really short pieces as he was so demanding of his compositions that every note had to be exactly right and there shouldn't be any "unnecessary" notes at all. I actually heard that he always wanted to write longer pieces and was always troubled by the fact that he just couldn't manage to do so, which in retrospect may actually be one of the great qualities of his work.

And Berg wrote rather short pieces too when he started to write 12-tone music as Schönberg's student, but Schönberg thwapped him on the head (well, not literally) as he thought Berg was trying to imitate Webern too much and he encouraged Berg to be more generous in his compositons and write as tonally as Berg felt the need to. And so Berg slowly started to write longer and longer pieces, partly of course also assisted by the fact that he wrote for operas (Wozzeck, Lulu), which gave him a formal/dramatical foundation to base his pieces on.

In fact, all the three composers of the Second Viennese School felt that songs, or any other pieces with a text as a foundation, were very helpful to them as a formal foundation for their pieces. This applies specifically to Schönberg in his pre-12-tone period. Pieces like "Pierrot Lunaire" or the "Hanging Gardens of Babylon" for example allowed him to compose atonally by ear without a strict system, but had a formal direction and coherence thanks to the connection to the text.

In a smaller scale, like in Schönberg's small piano pieces, such coherence mostly comes from intervallic relationships. In almost every piece of his atonical but not 12-tone period you'll find a dominance of certain intervals or interval constellations which hold the whole piece together. That may be for example a minor third and a minor second in various combinations and inversions in one piece, or (quite typical) combinations of tritoni with either fourths or fifths.

Another structural idea that is rather typical for the Second Viennese School are symmetric passages, in which for example a whole piece develops until the middle, followed by the exact retrograde. You'll find this a lot in Berg pieces, but of course you'll also find more refined variants of such techniques.

There are of course lots of other ways of connecting your material (if you want it to be connected), so be inventive. And SSC is of course very right that first, you should try to define what cohesiveness, form, stability, etc. mean to you.

But I perfectly know your pain. I find it incredibly hard to find the narrow border between an apparent "arbitrary" or even "random" form and an overly simplicistic one.