Do you mean an auto-cast effect, meaning it will only cast on a target one, but you can manually cast twice?
Or do you mean only allow a single cast on a particular target?
And where is the exclusion coming from? caster or target?
In the most likely case you need to create a validator that is true only if the buff that ability applies is not on the target. If it is, it will fail and prevent the ability cast.
@JohnPhobia: Goif rana is right the buff validator solution is the best one. BUT if you want several units to be able to cast the same ability on the target, but just prevent double cast from one caster things become dirty. you then have to work with markers (since you cannot easily validate the origin of a buff which is something i want for a long time already). and instead (or in combination with ) of using a buff, you have to use a persistant effect. the marker of the persistant has to be the same marker of your first effect of the ability and then validate with "no marker". you might need 2 effects in a row on your ability with the same marker ( no idea why, set effect worked for me)
I am pretty sure the untargetable flag is extremely wrong, due to it effecting many other targeting effects.
If it doesn't matter who casts it, then you want to add a buff to the target that lasts 5 seconds, and use the validator to disallow that ability on targets with that buff.
As others have noted, the best option is to place a hidden behavior on the targeted unit, and have the root/1st effect of the ability have a validator checking for the presence of that buff, and prevent casting if the behavior is present.
Ranakastrasz is correct that the untargetable flag will also affect all other abilities.
What you want to do is have this ability apply a behavior to the unit when it hits that unit. That behavior should have the 5 second duration that you mentioned. Then create a validator on behavior count on target unit, checking that this behavior is equal to 0.
Finally, find the absolute first effect of your ability, the one that you put directly into the ability data. Put this validator onto that effect. The reason for this is that the game will not even allow you to attempt to cast the ability if the validators on the first effect fail.