Your mystery grower is the gopher plant, Euphorbia lathyris. Some people plant it in their gardens in the hope that it will repel gophers and/or moles. (Unfortunately, that doesn’t work.) Be cautious and wear gloves when you work near it or remove it; its milky sap may cause skin irritation for some people and, worse yet, seriously damages eyes!
No, it’s not invasive. But if you allow it to drop its seeds, it’s likely you’ll have many more next year. I suggest you put on some protective gloves, then get rid of it now.
Read more about it here: https://en.wikipedia.org/wiki/Euphorbia_lathyris