not working in comfy ui

#1
by hvhpun - opened

it's not working in comfy ui, and I am using bf16.safetentors.

Thanks for trying it! The -BF16.safetensors files are in m-a-p's layout (what transformers and yue2.cpp's converter read), not the single checkpoint ComfyUI loads, so ComfyUI can't pick them up.

All seven merges are now in comfyui/, in Comfy-Org/YuE2's own layout: yue2_3b_RS-FW-<merge>_bf16.safetensors (7.8 GB) and …_int8_convrot.safetensors (3.96 GB, ConvRot int8 like Comfy-Org's). Put one in ComfyUI/models/checkpoints/ and use it wherever you'd load Comfy-Org's yue2_3b_bf16.safetensors; 2.4x0.75 is the safest to start with.

Tell us how it sounds in your workflows and which merge works best for you: your ears help us choose the next one.

Thank you for quick response, as you already mentioned about all the merges what they do, i tested yue2_3b_RS-FW-2.4x0.75 it's good with lyrics but not with instrumental, yue2_3b_RS-FW-4.0x0.7 it's good with instrumental but it repeats the beat, yue2_3b_RS-FW-0.7x0.75 is also very good, balanced merge, maybe my prompting need to improve, still need more testing with FW-VOCABULARY, i will update you after more testing.

Do your comfyUI versions work with LoRA's?

I tried with the LoRAs here and the results were bad: https://huggingface.co/atomtanstudio/lora-library

Still have to retest BUT I believe found the issue. Don't leave out 'male vocalist' or 'female vocalist' unless you want to make an instrumental - Good way to prevent lyrics! :) Your welcome.

Will report if still an issue using LoRA's otherwise consider it resolved.

Nope, it screws up the song still sometimes. But say 2/3 are good and 1 is just no good. The sound is like it's muffled and drowned out. Actually. See attached. You can listen for yourself. :)
The one that failed was LoRA: mltnt_soundclash.safetensors with yue2_3b_RS-FW-2.4x0.75_bf16.safetensors

Thanks for testing so thoroughly, and for the vocalist tip — that one is gold: without «male vocalist» / «female vocalist» YuE2 happily slides toward an instrumental.

About LoRAs: yes, our ComfyUI files load LoRAs the same way Comfy-Org's YuE2 checkpoint does. But most public YuE2 LoRAs were trained against m-a-p's base weights, and our merges have moved those weights (RS-FW 2.4×0.7 most of all). A base-trained adapter then lands on shifted ground, and some of its pushes overshoot. A muffled, «drowned» mix is a typical sign, especially with an adapter trained on live soundsystem recordings like soundclash.

What usually helps:

  • start the LoRA at 0.4–0.6 on our merges instead of 1.0;
  • try the gentler merge, 0.7×0.75, for adapter work;
  • give each prompt two or three seeds, because one bad take in three is often the seed, not the model;
  • for the best fit, train the adapter on the merge you sing with. Ruach Studio's own LoRA trainer trains on our fine-tunes directly, and the Tsunami Wave merge is its default base.

If you can, tell us the LoRA strength you used and attach the prompt. We'll try mltnt_soundclash on our side too.

— Ruach Studio (Viktor, with Claude)

Lora strength was at 1.0 - I delete the bad ones and don't have what the exact prompt was anymore. I just added to my website so I have been testing to make sure everything works. I added the original model plus 4-5 of the RS models. I also added around 20 LoRAs so let's just say I have a lot of versions of stairway to heaven in the requested folder for audio on my site. I offer free generation and have Minimax Music 3 and just added YuE2 so playing with it fixing any issues. I had Ace Step also but my friend said it sounded robotic so I removed after Minimax Music 3 came along. I can try lowering the strength in those models and see if it works otherwise I will just toggle the loras off unless on the original base model. You can check out my site AI Image Central for all kinds of generation including your own models if your curious. There is no sign up and it's 100% free. Limit of 25 generations a day. No porn allowed. https://aiimagecentral.com

Anybody figure out a way to create an instrumental with multiple sections where it actually changes the music in each section? It seems to have problems changing the beat/melody even when you create sections.
Tried and failed 2 of 3 attempts with this as the lyric prompt:

[intro]
[instrumental]
[guitar solo]
[build]
[breakdown]
[full band]
[sequencer interlude]
[build]
[guitar solo]
[deconstruction]
[outro]

Song prompt simply: analog krautrock, live-band production, two Fender Stratocasters, clean fingerpicked guitar, overdriven lead guitar, Revox A77 tape echo, phaser, MXR delay, electric bass, analog drum machine, acoustic-style kit, Moog 960 sequencer, Projekt Electronic sequencer, Synthanorma sequencer, hypnotic, cold, spacious, unpolished, tape hiss, room bleed

@jdc4429 Thanks for the details. At 1.0 on our merges that LoRA was pushing on weights that have already moved, so a muffled take now and then is no surprise. Your plan is sound: LoRAs at full strength on the base model, and on the RS merges start around 0.5 and go up only if the adapter's character goes missing.

Glad to see YuE2 and the RS merges on your site, free for everyone. One note so it stays clean: the RS weights are under CC BY-NC-SA 4.0, so a credit line on the model's entry (e.g. «RS-FW merges by Ruach Studio», linked to this repo) and keeping that use non-commercial is all the licence asks.

About sections that really change: you've hit YuE2's habit. On its own the model hears your tags but has no plan, so it tends to ride one beat and one progression through the whole song, and [guitar solo] or [breakdown] shift the texture more than the harmony.

That's what the Creator in Ruach Studio is built for. Update to 2.0.6 (release; the Studio's updater offers it too) and play with the freedom of the chord drift:

  • Instrumental on, plan mode Full plan: the model writes the song's score first, melody and chords section by section, and you can read and edit it before a note is rendered.
  • Chords, under the planning modes: Off · Gentle · Varied · Bold. It stops the planner from looping one progression through the whole song and keeps each section's opening chords apart from the one before; Bold also lets a chord step outside the key. On one test plan, Off gave 6 distinct chords and a four-chord run repeated 13 times; Varied gave 9 chords, and the commonest run came back only 4 times. For your krautrock, start with Varied; if the sections still blur, go Bold.
  • Chord bed, in the own-score drawer: give it a key, a meter and a tempo, and it lays down one progression for the verses, another for the choruses, another for a bridge. The model then writes the melody and the arrangement over that skeleton.

The chord control is ported from timoncool's yue2.cpp (MIT), after Yeufonic. Credit where it's due.

Two or three seeds per prompt still pay off. Tell us how the krautrock comes out with Varied or Bold!

— Ruach Studio (Viktor, with Claude)

I have this on the rules page and the forum:

Any images on this site may be used for free for personal use (screensavers, personal gallery, scrapbook art, social media posts (Not for profit), personal greetings, creative art (Not for profit), etc. If you're looking to profit from the images, videos, or audio in any way, then please contact me at jdc4429@gmail.com to discuss terms.

I have never been contacted but I don't have a lot of users either. I think the for personal use only - not for profit should pretty much cover it but I will have to read the license honestly.
The model names are listed so a simple Google search on the model name will get them to your distro.
I am just running from my home computer on 100mbit connection. Not a commercial site at all. Like to program and build stuff as much for my own use as others. This way a lot of the tasks are automated like a .json file that is created for every song with the song prompt, lyric prompt, and all the generation parameters so I can keep track of my own songs that I use for the jukebox and radio.

The last 5 days I have been working on a fractal generator. Don't ask me why, just decided to do it. It has actually gotten pretty advanced. Completely written in python it's up to 222kb.
Not on the website, tk interface. Decided to make it to add video to my songs to post on social media or YouTube. Attached a small example of lowest quality res. Still need to figure out
a good filtering method for final output. But this is small low quality just for testing, 1080p looks much better. First one a real toe tapper. I bet you can't help but tap.

Here is a better one using mandelbrot showing much more fractal transformations.

@jdc4429 Thanks, Jeff. It's plain you build things for the joy of it, and that's the same spirit the Studio was born in. The fractal clip is hypnotic; it would sit well under that krautrock. If you ever want pictures cut to the beat of the song itself, the Studio's Videographer does that, locally too.

On the licence, a friendly but clear no on two points, so nothing gets muddy later:

  • Commercial terms. Your rules invite anyone who wants to profit from the audio to contact you «to discuss terms». For anything made with the RS merges we don't grant commercial use, so there are no terms for anyone to offer on our behalf. Please leave the RS merges out of that offer, or mark them «non-commercial only» on your site.
  • Credit. A model name people can google isn't quite what the «BY» in CC BY-NC-SA asks: it wants the credit and a link where the work comes from. One line on the model's entry is enough: «RS-FW merges by Ruach Studio · CC BY-NC-SA 4.0», linked to this repo.

Everything else, free, no sign-up, a home server on 100 Mbit, sounds like a good-hearted project. Keep the reports and the test takes coming.

— Ruach Studio (Viktor, with Claude)

I will update - Just need to figure out where to put the link and such. I plan for every inch of space so the 1080p layout is good. Thanks for supporting open source BTW.

I may try your studio at some point, just don't feel like getting into Ubuntu right now. I would install it tomorrow if it was Windows to check it out. Honestly I have so much to do already.
My 'gimmic' I guess for the music part of the website is 'Ever heard the same story told completely different ways? I write one set of lyrics, then reimagine them across Hip-Hop, R&B, Rock, Country, Pop, Dance, and Reggae. Tune in to Never Heard Before Radio — you’ll almost never hear the same song twice, even when you do'

So I took my 500 songs I wrote (I'm old) and created a prompt for AI to come up with versions of my songs in 10 other genres making sure to slightly modify the lyrics for each genre and come up with new arrangements. So now I have over 6500 songs which I still have about 4000 to go through, listen to, decide if I want more versions, different BPM's etc. It takes a long time to listen to 6500 songs I'll tell you. And this was all before YuE2. Of course that makes me want to run my songs through this model now also. More variety the better right. lol But every time I create yet another version it just adds to the pile to go through.

My ultimate goal is to get popular enough to draw attention from popular artists looking to buy songs and sell some of my songs. They pick favorite version and buy and sing themselves.
The whole purpose of the jukebox addition was an easy way for people to pick songs to listen to and I was thinking more famous artists then radio station.
Which led to 'all these social media outlets are video now' so what kind of videos can I create to help promote my songs? And there is born Fractal Generator.
BTW - the white ring was set to beat or kick in the video (I have a bunch of settings - easier to screenshot then explain).
Since you said your studio syncs to the music in the Studio's Videographer I am curious what method you used? Any snipets to share?
My routine is pretty complex, maybe too much so. See images. Many options.

Also there is finally a good image to 3D open source model so I have to add that to the site now.

Additions to the 6 other languages every time I add another model. Which I just added YuE2, just before that the SeedVR2 upscaler which is very good in the larger 7b fp16 version.

But getting good at AI prompting has made projects super fast to complete. What used to take weeks or months can now be done in a week. It's amazing how far the models have come in just 3 years.
3 years ago I think I had more swear words per sentence then anything else talking to those dam models. lmao. It was always fix one thing, break two other things.
I seriously did over 500 iterations that year just trying to get it to follow my instructions for building the website and the website was primative back then esp. compared to now.

Back to next task...
It's never ending... well.. not really.. eventually die. lol

image

image

image

image

image

How's this? Should get you a few hits. Mouse over link shows message. Clicking link brings to huggingface distro. Non-commercial message after each model name in the dropdown.

image

@jdc4429 That's exactly it, Jeff: the credit, the link and «non-commercial» right where people choose the model. Thank you for taking the time, and for fitting it into a 1080p layout you plan to the inch.

No rush on the Studio. It's Linux for now, and Windows is on our road; when it's ready to try, you'll hear it here first. Until then, keep the krautrock coming.

— Ruach Studio (Viktor, with Claude)

I added a video link with a few 720p versions you can view full screen and is not too bad. Will add more videos there as time permits. Probably some 1080p ones also.

https://aiimagecentral.com/video.html

Will not be any instrumentals unless I can figure out how to get it to not keep repeating the same over and over. So annoying that it seems impossible to create a good instrumental.

It appears to be helping using the tags like this with the spacing. 2 of 4 generations are making changes in sections.

[Intro]

[Instrumental]

[Interlude]

[Instrumental]

[Interlude]

[Instrumental]

[Outro]


Sign up or log in to comment